docs: overhaul documentation

README.md

@@ -240,31 +240,19 @@ Tool instructions go here.
 
 Tool parameters are key-value pairs defined at the beginning of a tool block, before any instructional text. They are specified in the format `key: value`. The parser recognizes the following keys (case-insensitive and spaces are ignored):
 
-`Name`: The name of the tool.
 
-`Model Name`: The OpenAI model to use, by default it uses "gpt-4-turbo-preview"
+| Key            | Description                                                                                                                             |
+|------------------|-----------------------------------------------------------------------------------------------------------------------------------------|
+| `Name`           | The name of the tool.                                                                                                                   |
+| `Model Name`     | The OpenAI model to use, by default it uses "gpt-4-turbo-preview"                                                                         |
+| `Description`    | The description of the tool. It is important that this properly describes the tool's purpose as the description is used by the LLM.     |
+| `Internal Prompt`| Setting this to `false` will disable the built-in system prompt for this tool.                                                          |
+| `Tools`          | A comma-separated list of tools that are available to be called by this tool.                                                            |
+| `Args`           | Arguments for the tool. Each argument is defined in the format `arg-name: description`.                                                  |
+| `Max Tokens`     | Set to a number if you wish to limit the maximum number of tokens that can be generated by the LLM.                                      |
+| `JSON Response`  | Setting to `true` will cause the LLM to respond in a JSON format. If you set true you must also include instructions in the tool.        |
+| `Temperature`    | A floating-point number representing the temperature parameter. By default, the temperature is 0. Set to a higher number for more creativity. |
 
-`Description`: The description of the tool. It is important that the properly describes the tool's purpose as the
-description is used by the LLM to determine if the tool is to be invoked.
-
-`Internal Prompt`: Setting this to `false` will disable the built in system prompt for this tool. GPTScript includes a
-default system prompt to instruct the AI to behave more like a script engine and not a "helpful assistant."
-
-`Tools`: A comma-separated list of tools that are available to be called by this tool. A tool can only call the tools
-that are defined here. A tool name can reference a name in the current file, or a file relative to the current directory
-of the tool file, a http(s) URL, or a file in GitHub using github.com/username/repo/file.gpt format. When referencing
-a file or URL the tool loaded is the first tool in the file. To reference a specific tool in a file or URL use the
-syntax `tool-name from file-or-url`.
-
-`Args`: Arguments for the tool. Each argument is defined in the format `arg-name: description`. All arguments are essentially
-strings. No other type really exists as all input and output to tools is text based.
-
-`Max Tokens`: Set to a number if you wish to limit the maximum number of tokens that can be generated by the LLM.
-
-`JSON Response`: Setting to `true` will cause the LLM to respond in a JSON format. If you set true you must also include instructions in the tool
-to inform the LLM to respond in some JSON structure.
-
-`Temperature`: A floating-point number representing the temperature parameter. By default the temperature is 0. Set to a higher number to make the LLM more creative.
 
 #### Tool Body
 

docs/docs/03-tools/02-authoring.md

@@ -0,0 +1,87 @@
+# Authoring Tools
+
+You can author your own tools for your use or to share with others.
+The process for authoring a tool is as simple as creating a `tool.gpt` file in the root directory of your project.
+This file is itself a GPTScript that defines the tool's name, description, and what it should do.
+
+## Quickstart
+
+This is a guide for writing portable tools for GPTScript. The supported languages currently are Python, NodeJS, and Go. This guide uses Python but you can see documentation for the other language below.
+
+### 1. Write the code
+
+Create a file called `tool.py` with the following contents:
+
+```python
+import os
+import requests
+
+print(requests.get(os.getenv("url")).text)
+```
+
+Create a file called `requirements.txt` with the following contents:
+
+```
+requests
+```
+
+### 2. Create the tool
+
+Create a file called `tool.gpt` with the following contents:
+
+```
+Description: Returns the contents of a webpage.
+Args: url: The URL of the webpage.
+
+#!/usr/bin/env python3 ${GPTSCRIPT_TOOL_DIR}/tool.py
+```
+
+:::tip
+Every arg becomes an environment variable when the tool is invoked. So instead of accepting args using flags like `--size="${size}", your program can just read the `size` environment variable.
+:::
+
+The `GPTSCRIPT_TOOL_DIR` environment variable is automatically populated by GPTScript so that the tool
+will be able to find the `tool.py` file no matter what the user's current working directory is.
+
+If you make the tool available in a public GitHub repo, then you will be able to refer to it by
+the URL, i.e. `github.com/<user>/<repo name>`. GPTScript will automatically set up a Python virtual
+environment, install the required packages, and execute the tool.
+
+### 3. Use the tool
+
+Here is an example of how you can use the tool once it is on GitHub:
+
+```
+Tools: github.com/<user>/<repo name>
+
+Get the contents of https://github.com
+```
+
+## Sharing Tools
+
+GPTScript is designed to easily export and import tools. Doing this is currently based entirely around the use of GitHub repositories. You can export a tool by creating a GitHub repository and ensureing you have the `tool.gpt` file in the root of the repository. You can then import the tool into a GPTScript by specifying the URL of the repository in the `tools` section of the script. For example, we can leverage the `image-generation` tool by adding the following line to a GPTScript:
+
+```yaml
+tools: github.com/gptscript-ai/image-generation
+
+Generate an image of a city skyline at night.
+```
+
+### Supported Languages
+
+GPTScript can execute any binary that you ask it to. However, it can also manage the installation of a language runtime and dependencies for you. Currently this is only supported for a few languages. Here are the supported languages and examples of tools written in those languages:
+
+| Language | Example |
+|----------|---------|
+| `Python`   | [Image Generation](https://github.com/gptscript-ai/image-generation) - Generate images based on a prompt |
+| `Node.js`  | [Vision](https://github.com/gptscript-ai/vision) - Analyze and interpret images |
+| `Golang`   | [Search](https://github.com/gptscript-ai/search) - Use various providers to search the internet |
+
+
+### Automatic Documentation
+
+Each GPTScript tool is self-documented using the `tool.gpt` file. You can automatically generate documentation for your tools by visiting `tools.gptscript.ai/<github repo url>`. This documentation site allows others to easily search and explore the tools that have been created. 
+
+You can add more information about how to use your tool by adding an `examples` directory to your repository and adding a collection of `.gpt` files that demonstrate how to use your tool. These examples will be automatically included in the documentation.
+
+For more information and to explore existing tools, visit [tools.gptscript.ai](https://tools.gptscript.ai).

docs/docs/04-use-cases.md

@@ -0,0 +1,91 @@
+# Use Cases
+
+## Retrieval
+
+Retrieval-Augmented Generation (RAG) leverages a knowledge base outside of the training data before consulting the LLM to generate a response.
+The following GPTScript implements RAG:
+
+```yaml
+name: rag
+description: implements retrieval-augmented generation
+args: prompt: a string
+tools: query
+
+First query the ${prompt} in the knowledge base, then construsts an answer to ${prompt} based on the result of the query.
+
+---
+
+name: query
+description: queries the prompt in the knowledge base
+args: prompt: a string
+
+... (implementation of knowledge base query follows) ...
+```
+
+The idea behind RAG is simple. Its core logic can be implemented in one GPTScript statement: `First query the ${prompt} in the knowledge base, then construsts an answer to ${prompt} based on the result of the query.` The real work of building RAG lies in the tool that queries your knowledge base.
+
+You construct the appropriate query tool based on the type of knowledge base you have.
+
+| Knowledge Base | Query Tool |
+|------|------|
+| A vector database storing common document types such as text, HTML, PDF, and Word | Integrate with LlamaIndex for vector database support [Link to example]|
+| Use the public or private internet to supply up-to-date knowledge | Implement query tools using a search engine, as shown in [`search.gpt`](https://github.com/gptscript-ai/gptscript/tree/main/examples/search.gpt)|
+| A structured database supporting SQL such as sqlite, MySQL, PostgreSQL, and Oracle DB | Implement query tools using database command line tools such as `sqlite` and `mysql` [Link to example]|
+| An ElasticSearch/OpenSearch database storing logs or other text files | Implement query tools using database command line tools [Link to example]|
+| Other databases such as graph or time series databases | Implement query tools using database command line tools [Link to example]|
+
+## Task Automation
+
+### Planning
+
+Here is a GPTScript that produces a detailed travel itinerary based on inputs from a user: [`travel-agent.gpt`](https://github.com/gptscript-ai/gptscript/tree/main/examples/travel-agent.gpt)
+
+### Web UI Automation
+
+Here is a GPTScript that automates a web browser to browse and navigate website, extract information: [coachella-browse.gpt](https://github.com/gptscript-ai/browser/tree/main/examples/coachella-browse.gpt)
+
+For additional examples, please explore [here](https://github.com/gptscript-ai/browser?tab=readme-ov-file#examples).
+
+### API Automation
+
+Here are a few GPTScripts that interact with issues on GitHub:
+
+- [Create GitHub Issues](https://github.com/gptscript-ai/create-github-issues/tree/main/examples/example.gpt)
+- [Modify GitHub Issues](https://github.com/gptscript-ai/modify-github-issues/tree/main/examples/example.gpt)
+- [Query GitHub Issues](https://github.com/gptscript-ai/gptscriptquery-github-issues/tree/main/examples/example.gpt)
+
+## Agents and Assistants
+
+Agents and assistants are synonyms. They are software programs that leverage LLM to carry out tasks.
+
+In GPTScript, agents and assistants are implemented using tools. Tools can use other tools. Tools can be implemented using natural language prompts or using traditional programming languages such as Python or JavaScript. You can therefore build arbitrarily complex agents and assistants in GPTScript. Here is an example of an assistant that leverages an HTTP client, MongoDB, and Python code generation to display Hacker News headlines: [`hacker-news-headlines.gpt`](https://github.com/gptscript-ai/gptscript/tree/main/examples/hacker-news-headlines.gpt)
+
+## Data Analysis
+
+Depending on the context window supported by the LLM, you can either send a large amount of data to the LLM to analyze in one shot or supply data in batches.
+
+### Summarization
+
+Here is a GPTScript that sends a large document in batches to the LLM and produces a summary of the entire document. [hamlet-summarizer](https://github.com/gptscript-ai/gptscript/tree/main/examples/hamlet-summarizer)
+
+### Tagging
+
+Here is a GPTScript that performs sentiment analysis on the input text: [Social Media Post Sentiment Analyzer](https://github.com/gptscript-ai/gptscript/tree/main/examples/sentiments)
+
+### CSV Files
+
+Here is a GPTScript that reads the content of a CSV file and make query using natural language: [csv-reader.gpt](https://github.com/gptscript-ai/csv-reader/tree/main/examples/csv-reader.gpt)
+
+### Understanding Code
+
+Here is a GPTScript that summarizes the code stored under the current directory: [`describe-code.gpt`](https://github.com/gptscript-ai/gptscript/tree/main/examples/describe-code.gpt)
+
+## Vision, Image, and Audio
+
+### Vision
+
+Here is an example of a web app that leverages GPTScript to recognize ingredients in a photo and suggest a recipe based on them: [recipe-generator](https://github.com/gptscript-ai/gptscript/tree/main/examples/recipegenerator).
+
+### Image Generation
+
+Here is a GPTScript that takes a story prompt and generates an illustrated children's book: [story-book.gpt](https://github.com/gptscript-ai/gptscript/tree/main/examples/story-book)

docs/docs/05-how-it-works.md

@@ -0,0 +1,61 @@
+# How it works
+
+**_GPTScript is composed of tools._** Each tool performs a series of actions similar to a function. Tools have available
+to them other tools that can be invoked similar to a function call. While similar to a function, the tools are
+primarily implemented with a natural language prompt. **_The interaction of the tools is determined by the AI model_**,
+the model determines if the tool needs to be invoked and what arguments to pass. Tools are intended to be implemented
+with a natural language prompt but can also be implemented with a command or HTTP call.
+
+## Example
+
+Below are two tool definitions, separated by `---`. The first tool does not require a name or description, but
+every tool after name and description are required. The first tool, has the parameter `tools: bob` meaning that the tool named `bob` is available to be called if needed.
+
+```yaml
+tools: bob
+
+Ask Bob how he is doing and let me know exactly what he said.
+
+---
+name: bob
+description: I'm Bob, a friendly guy.
+args: question: The question to ask Bob.
+
+When asked how I am doing, respond with "Thanks for asking "${question}", I'm doing great fellow friendly AI tool!"
+```
+
+Put the above content in a file named `bob.gpt` and run the following command:
+
+```shell
+$ gptscript bob.gpt
+```
+
+```
+OUTPUT:
+
+Bob said, "Thanks for asking 'How are you doing?', I'm doing great fellow friendly AI tool!"
+```
+
+Tools can be implemented by invoking a program instead of a natural language prompt. The below
+example is the same as the previous example but implements Bob using python.
+
+```yaml
+Tools: bob
+
+Ask Bob how he is doing and let me know exactly what he said.
+
+---
+Name: bob
+Description: I'm Bob, a friendly guy.
+Args: question: The question to ask Bob.
+
+#!python3
+
+import os
+
+print(f"Thanks for asking {os.environ['question']}, I'm doing great fellow friendly AI tool!")
+```
+
+With these basic building blocks you can create complex scripts with AI interacting with AI, your local system, data,
+or external services.
+

docs/docs/06-gpt-file-reference.md

@@ -0,0 +1,82 @@
+# GPT File Reference
+
+## Extension
+
+GPTScript files use the `.gpt` extension by convention.
+
+## File Structure
+
+A GPTScript file has one or more tools in the file. Each tool is separated by three dashes `---` alone on a line.
+
+```yaml
+Name: tool1
+Description: This is tool1
+
+Do sample tool stuff.
+
+---
+Name: tool2
+Description: This is tool2
+
+Do more sample tool stuff.
+```
+
+## Tool Definition
+
+A tool starts with a preamble that defines the tool's name, description, args, available tools and additional parameters.
+The preamble is followed by the tool's body, which contains the instructions for the tool. Comments in
+the preamble are lines starting with `#` and are ignored by the parser. Comments are not really encouraged
+as the text is typically more useful in the description, argument descriptions or instructions.
+
+```yaml
+Name: tool-name
+# This is a comment in the preamble.
+Description: Tool description
+# This tool can invoke tool1 or tool2 if needed
+Tools: tool1, tool2
+Args: arg1: The description of arg1
+
+Tool instructions go here.
+```
+
+## Tool Parameters
+
+Tool parameters are key-value pairs defined at the beginning of a tool block, before any instructional text. They are specified in the format `key: value`. The parser recognizes the following keys (case-insensitive and spaces are ignored):
+
+| Key            | Description                                                                                                                             |
+|------------------|-----------------------------------------------------------------------------------------------------------------------------------------|
+| `Name`           | The name of the tool.                                                                                                                   |
+| `Model Name`     | The OpenAI model to use, by default it uses "gpt-4-turbo-preview"                                                                         |
+| `Description`    | The description of the tool. It is important that this properly describes the tool's purpose as the description is used by the LLM.     |
+| `Internal Prompt`| Setting this to `false` will disable the built-in system prompt for this tool.                                                          |
+| `Tools`          | A comma-separated list of tools that are available to be called by this tool.                                                            |
+| `Args`           | Arguments for the tool. Each argument is defined in the format `arg-name: description`.                                                  |
+| `Max Tokens`     | Set to a number if you wish to limit the maximum number of tokens that can be generated by the LLM.                                      |
+| `JSON Response`  | Setting to `true` will cause the LLM to respond in a JSON format. If you set true you must also include instructions in the tool.        |
+| `Temperature`    | A floating-point number representing the temperature parameter. By default, the temperature is 0. Set to a higher number for more creativity. |
+
+
+
+## Tool Body
+
+The tool body contains the instructions for the tool which can be a natural language prompt or
+a command to execute. Commands must start with `#!` followed by the interpreter (e.g. `#!/bin/bash`, `#!python3`)
+a text that will be placed in a file and passed to the interpreter. Arguments can be references in the instructions
+using the format `${arg1}`.
+
+```yaml
+name: echo-ai
+description: A tool that echos the input
+args: input: The input
+
+Just return only "${input}"
+
+---
+name: echo-command
+description: A tool that echos the input
+args: input: The input
+
+#!/bin/bash
+
+echo "${input}"
+```