feat: refactor docs and add content around tools and authoring

- chore: add note about adding local model support
- docs: add image-generator cookbook
- docs: add gen-docs.gpt and a make target for it

Signed-off-by: tylerslaton <[email protected]>

Author: tylerslaton

Makefile

@@ -39,5 +39,8 @@ validate: tidy lint
 ci: build
 	./bin/gptscript ./scripts/ci.gpt
 
+gen-docs: build
+	./bin/gptscript ./scripts/gen-docs.gpt
+
 serve-docs:
-	(cd docs && npm start)
+	(cd docs && npm i && npm start)

README.md

@@ -2,11 +2,14 @@
 
 ## Overview
 
-GPTScript is a new scripting language to automate your interaction with a Large Language Model (LLM), namely OpenAI. The ultimate goal is to create a fully natural language based programming experience. The syntax of GPTScript is largely natural language, making it very easy to learn and use.
+GPTScript is a new scripting language to automate your interaction with a Large Language Model (LLM), namely OpenAI. The ultimate goal is to create a natural language programming experience. The syntax of GPTScript is largely natural language, making it very easy to learn and use.
 Natural language prompts can be mixed with traditional scripts such as bash and python or even external HTTP service
-calls. With GPTScript you can do just about anything like [plan a vacation](./examples/travel-agent.gpt),
+calls. With GPTScript you can do just about anything, like [plan a vacation](./examples/travel-agent.gpt),
 [edit a file](./examples/add-go-mod-dep.gpt), [run some SQL](./examples/sqlite-download.gpt), or [build a mongodb/flask app](./examples/hacker-news-headlines.gpt).
 
+| :memo: | We are currently exploring options for interacting with local models using GPTScript. |
+|-|:-|
+
 ```yaml
 # example.gpt
 
@@ -59,10 +62,18 @@ Download and install the archive for your platform and architecture from the [re
 
 ### 2. Get an API key from [OpenAI](https://platform.openai.com/api-keys).
 
+#### macOS and Linux
+
 ```shell
 export OPENAI_API_KEY="your-api-key"
 ```
 
+#### Windows
+
+```powershell
+$env:OPENAI = 'your-api-key'
+```
+
 ### 3. Run Hello World
 
 ```shell
@@ -175,7 +186,7 @@ Description: Tool description
 # This tool can invoke tool1 or tool2 if needed
 Tools: tool1, tool2
 Args: arg1: The description of arg1
-      
+
 Tool instructions go here.
 ```
 #### Tool Parameters
@@ -228,7 +239,7 @@ description: A tool that echos the input
 args: input: The input
 
 #!/bin/bash
-        
+
 echo "${input}"
 ```
 
@@ -247,7 +258,7 @@ Join us on Discord: [![Discord](https://img.shields.io/discord/12045584209848648
 
 ## License
 
-Copyright (c) 2023 [Acorn Labs, Inc.](http://acorn.io)
+Copyright (c) 2024 [Acorn Labs, Inc.](http://acorn.io)
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

docs/docs/01-overview.md

@@ -0,0 +1,41 @@
+---
+title: Overview
+slug: /
+---
+
+GPTScript is a new scripting language to automate your interaction with a Large Language Model (LLM), namely OpenAI. The ultimate goal is to create a natural language programming experience. The syntax of GPTScript is largely natural language, making it very easy to learn and use. Natural language prompts can be mixed with traditional scripts such as bash and python or even external HTTP service calls. With GPTScript you can do just about anything, like [plan a vacation](https://github.com/gptscript-ai/gptscript/blob/main/examples/travel-agent.gpt), [edit a file](https://github.com/gptscript-ai/gptscript/blob/main/examples/add-go-mod-dep.gpt), [run some SQL](https://github.com/gptscript-ai/gptscript/blob/main/examples/sqlite-download.gpt), or [build a mongodb/flask app](https://github.com/gptscript-ai/gptscript/blob/main/examples/hacker-news-headlines.gpt).
+
+:::note
+We are currently exploring options for interacting with local models using GPTScript.
+:::
+
+```yaml
+# example.gpt
+
+Tools: sys.download, sys.exec, sys.remove
+
+Download https://www.sqlitetutorial.net/wp-content/uploads/2018/03/chinook.zip to a
+random file. Then expand the archive to a temporary location as there is a sqlite
+database in it.
+
+First inspect the schema of the database to understand the table structure.
+
+Form and run a SQL query to find the artist with the most number of albums and output
+the result of that.
+
+When done remove the database file and the downloaded content.
+```
+```
+$ gptscript ./example.gpt
+
+OUTPUT:
+
+The artist with the most number of albums in the database is Iron Maiden, with a total
+of 21 albums.
+```
+
+For more examples check out the [examples](https://github.com/gptscript-ai/gptscript/blob/main/examples) directory.
+
+## Community
+
+Join us on Discord: [![Discord](https://img.shields.io/discord/1204558420984864829?label=Discord)](https://discord.gg/9sSf4UyAMC)

docs/docs/02-getting-started.md

@@ -0,0 +1,61 @@
+# Getting Started
+
+### 1. Install the latest release
+
+#### Homebrew (macOS and Linux)
+
+```shell
+brew install gptscript-ai/tap/gptscript
+```
+
+#### Install Script (macOS and Linux):
+
+```shell
+curl https://get.gptscript.ai/install.sh | sh
+```
+
+#### WinGet (Windows)
+
+```shell
+winget install gptscript-ai.gptscript
+```
+
+#### Manually
+
+Download and install the archive for your platform and architecture from the [releases page](https://github.com/gptscript-ai/gptscript/releases).
+
+### 2. Get an API key from [OpenAI](https://platform.openai.com/api-keys).
+
+#### macOS and Linux
+
+```shell
+export OPENAI_API_KEY="your-api-key"
+```
+
+#### Windows
+
+```powershell
+$env:OPENAI = 'your-api-key'
+```
+
+### 3. Run Hello World
+
+```shell
+gptscript https://get.gptscript.ai/echo.gpt --input 'Hello, World!'
+
+OUTPUT:
+
+Hello, World!
+```
+The model used by default is `gpt-4-turbo-preview` and you must have access to that model in your OpenAI account.
+
+### 4. Extra Credit: Examples and Run Debugging UI
+
+Clone examples and run debugging UI
+```shell
+git clone https://github.com/gptscript-ai/gptscript
+cd gptscript/examples
+
+# Run the debugging UI
+gptscript --server
+```

docs/docs/100-tools/01-using.md

@@ -0,0 +1,53 @@
+# Using Tools
+In GPTScript, tools are used to extend the capabilities of a script. The idea behind them is that AI performs better when it has very specific instructions for a given task. Tools are a way to break-up the problem into smaller and more focused pieces where each tool is responsible for a specific task. A typical flow like this is to have a main script that imports a set of tools it can use to accomplish its goal.
+
+GPTScripts can utilize tools in one of three ways:
+1. Built-in system tools
+2. In-script tools
+3. External tools
+
+### System Tools
+All GPTScripts have access to system tools, like `sys.read` and `sys.write`, that can be used without any additional configuration.
+
+```yaml
+tools: sys.read, sys.write
+
+Read all of the files in my current directory, do not recurse over any subdirectories, and give me a description of this directory's contents.
+```
+
+System tools are a set of core tools that come packaged with GPTScript by default.
+
+### In-Script Tools
+Things get more interesting when you start to use custom tools.
+
+The most basic example of this is an in-script tool that is defined in the same file as the main script. This is useful for breaking up a large script into smaller, more manageable pieces.
+
+```yaml
+tools: random-number
+
+Select a number at random and, based on the results, write a poem about it.
+
+---
+name: random-number
+description: Generate a random number between 1 and 100.
+
+Select a number at random between 1 and 100 and return only the number.
+```
+
+### External Tools
+
+This is where the real power of GPTScript starts to become evident. For this example lets use the external [image-generation](https://github.com/gptscript-ai/image-generation) and [search](https://github.com/gptscript-ai/search) tools to generate an image and then search the web for similar images.
+
+:::note
+There will be better packaging and distribution for external tools in the future. For now, this assumes you have the tools cloned locally and are pointing to their repos directly.
+:::
+
+```yaml
+tools: ./image-generation/tool.gpt, ./vision/tool.gpt, sys.read
+
+Generate an image of a city skyline at night and write the resulting image to a file called city_skyline.png.
+
+Take this image and write a description of it in the style of pirate.
+```
+
+External tools are tools that are defined by a `tool.gpt` file in their root directory. They can be imported into a GPTScript by specifying the path to the tool's root directory.

docs/docs/100-tools/02-authoring.md

@@ -0,0 +1,112 @@
+# Authoring Tools
+
+You can author your own tools for your use or to share with others. The process for authoring a tool is as simple as creating a `tool.gpt` file in the root directory of your project. This file is itself a GPTScript that defines the tool's name, description, and what it should do.
+
+Here's an example of the `tool.gpt` file for the `image-generation` tool:
+
+```yaml
+description: I am a tool that can generate images based on arguments that are sent to me. I return a list of URLs to the generated images.
+args: prompt: (required) The text prompt based on which the GPT model will generate a response
+args: model: (optional) The model to use for image generation. Defaults to "dall-e-3".
+args: size: (optional) The size of the image to generate, format WxH (e.g. 1024x1024). Defaults to 1024x1024.
+args: quality: (optional) The quality of the generated image. Allowed values are "standard" or "hd". Default is "standard".
+args: number: (optional) The number of images to generate. Defaults to 1.
+
+#!/usr/bin/env python3 ./cli.py --prompt="${prompt}" --model="${model}" --size="${size}" --quality="${quality}" --number="${number}"
+```
+
+At the bottom you'll notice a shebang line that specifies the command to run when the tool is invoked. This is the exact command that will be executed when the tool is used in a GPTScript. Doing this with tools allows for a high degree of reliability that the tool would not otherwise have.
+
+:::tip
+Every arg becomes an environment variable when the tool is invoked.
+:::
+
+## Binary Tools
+GPTScript can call binaries and scripts located on your system or externally. This is done by defining a `tool.gpt` file in the same directory that the binary or script is located. In that `tool.gpt` file, you call it directly using the `#!` directive.
+
+For example:
+
+```yaml
+description: This is a tool that calls a binary.
+args: arg1: The first argument.
+
+#!/usr/bin/env ./path/to/binary --arg1="${arg1}"
+```
+
+## Shell
+You can call shell scripts directly using the `#!` directive. For example:
+
+```yaml
+description: This is a tool that calls a shell script.
+args: arg1: The first argument.
+
+#!/usr/bin/env sh
+echo "Hello, world!"
+./path/to/script.sh --arg1="${arg1}"
+```
+
+## Python
+You can call Python scripts directly, for example:
+
+```yaml
+description: This is a tool that calls a Python script.
+args: arg1: The first argument.
+
+#!/usr/bin/env python3 ./path/to/script.py --arg1="${arg1}"
+```
+
+If you need to bootstrap a Python environment, you can use a virtual environment. For example:
+
+```yaml
+description: This is a tool that calls a Python script in a virtual environment.
+args: arg1: The first argument.
+
+#!/usr/bin/env sh
+python3 -m venv .venv
+pip3 install -r requirements.txt
+python3 ./path/to/script.py --arg1="${arg1}"
+```
+
+## Node
+
+You can call Node.js scripts directly, for example:
+
+```yaml
+description: This is a tool that calls a Node.js script.
+args: arg1: The first argument.
+
+#!/usr/bin/env node ./path/to/script.js --arg1="${arg1}"
+```
+
+If you need to bootstrap a Node.js environment, you can use call npm in the `tool.gpt` file. For example:
+
+```yaml
+description: This is a tool that calls a Node.js script with a package manager.
+args: arg1: The first argument.
+
+#!/usr/bin/env sh
+npm install
+node ./path/to/script.js --arg1="${arg1}"
+```
+
+## Golang
+
+You can call Golang binaries directly, for example:
+
+```yaml
+description: This is a tool that calls a Golang binary.
+args: arg1: The first argument.
+
+#!/usr/bin/env ./path/to/binary --arg1="${arg1}"
+```
+
+If you need to bootstrap a Golang binary, you can the go CLI to build it before running it. For example:
+
+```yaml
+description: This is a tool that calls a Golang binary.
+args: arg1: The first argument.
+
+#!/usr/bin/env sh
+go build -o ./path/to/binary
+./path/to/binary --arg1="${arg1}"
+```

docs/docs/tools/image-generation.md renamed to docs/docs/100-tools/03-offerings/01-image-generation.md

@@ -1,23 +1,34 @@
 # Image Generation
 
-## Overview
-
 The `image-generation` tool leverages OpenAI's DALL-E model to convert textual descriptions into images. It offers a versatile approach to creating visual content, making it suitable for a wide range of applications, from content creation to design assistance.
 
 For more information, check out the tool [here](https://github.com/gptscript-ai/image-generation)
 
+## Installation
+We are working on an process where installation of tools will be as simple as
+referencing the tool's link in your GPTScript. For now, you can follow the steps below to install the image-generation tool.
+
+1. Clone the image-generation tool repository [from GitHub](https://github.com/gptscript-ai/image-generation).
+
+```shell
+git clone https://github.com/gptscript-ai/image-generation
+```
+
+2. Run `make bootstrap` to setup the python venv and install the required dependencies.
+
+```shell
+make bootstrap
+```
+
+3. Reference `<path-to-the-repo>/tool.gpt` in your GPTScript.
+
 ## Usage
 To use the Image Generation tool, you need to make sure that the OPENAI_API_KEY environment variable is set to your OpenAI API key. You can obtain an API key from the OpenAI platform if you don't already have one.
 
 With that setup, you can use it in any GPTScript like so:
 
-:::tip
-This script assumes you have the `image-generation` tool repo cloned locally and located at `./image-generation`. The ability to import tools from remote
-locations is coming soon.
-:::
-
 ```
-tools: ./image-generation, sys.write
+tools: ./image-generation/tool.gpt, sys.download, sys.write
 
 Generate an image of a city skyline at night with a resolution of 512x512 pixels.
 

docs/docs/tools/system.md renamed to docs/docs/100-tools/03-offerings/02-system.md

@@ -104,3 +104,12 @@ Writes content to a specified file.
 
 - `content`: The content to be written to the file.
 - `filename`: The filename where the content should be written.
+
+## sys.append
+
+Appends the contents to a file.
+
+#### Arguments
+
+- `content`: The content to append.
+- `filename`: The name of the file to append to.