From 6a93b9f1a3002d44062ddda91d0dbfe51ea10682 Mon Sep 17 00:00:00 2001 From: Evan Jones Date: Mon, 22 May 2023 23:38:52 -0400 Subject: [PATCH 1/2] readme : add docs for chat-persistent.sh --- README.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/README.md b/README.md index 102cde43fb457..c35882befc6aa 100644 --- a/README.md +++ b/README.md @@ -391,6 +391,28 @@ Note the use of `--color` to distinguish between user input and generated text. ![image](https://user-images.githubusercontent.com/1991296/224575029-2af3c7dc-5a65-4f64-a6bb-517a532aea38.png) +### Persistent Interaction + +The prompt, user inputs, and model generations can be saved and resumed across calls to `./main` by leveraging `--prompt-cache` and `--prompt-cache-all`. The `./examples/chat-persistent.sh` script demonstrates this with support for long-running, resumable chat sessions. To use this example, you must provide a file to cache the initial chat prompt and a directory to save the chat session, and may optionally provide the same variables as `chat-13B.sh`. The same prompt cache can be reused for new chat sessions. Note that both prompt cache and chat directory are tied to the initial prompt (`PROMPT_TEMPLATE`) and the model file. + +```bash +# Start a new chat +PROMPT_CACHE_FILE=chat.prompt.bin CHAT_SAVE_DIR=./chat/default \ + MODEL=~/llama-models/30B/ggml-model-q4_0. ./examples/chat-persistent.sh + +# Resume that chat +PROMPT_CACHE_FILE=chat.prompt.bin CHAT_SAVE_DIR=./chat/default \ + MODEL=~/llama-models/30B/ggml-model-q4_0. ./examples/chat-persistent.sh + +# Start a different chat with the same prompt/model +PROMPT_CACHE_FILE=chat.prompt.bin CHAT_SAVE_DIR=./chat/another \ + MODEL=~/llama-models/30B/ggml-model-q4_0. ./examples/chat-persistent.sh + +# Different prompt cache for different prompt/model +PROMPT_TEMPLATE=./prompts/chat-with-bob.txt PROMPT_CACHE_FILE=bob.prompt.bin \ + CHAT_SAVE_DIR=./chat/bob MODEL=~/llama-models/30B/ggml-model-q4_0. ./examples/chat-persistent.sh +``` + ### Instruction mode with Alpaca 1. First, download the `ggml` Alpaca model into the `./models` folder From ba000f99418502e85180ede6a8924e0c0c953f85 Mon Sep 17 00:00:00 2001 From: Evan Jones Date: Mon, 22 May 2023 23:46:30 -0400 Subject: [PATCH 2/2] Update README.md --- README.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index c35882befc6aa..f88e520ee4f2c 100644 --- a/README.md +++ b/README.md @@ -397,20 +397,17 @@ The prompt, user inputs, and model generations can be saved and resumed across c ```bash # Start a new chat -PROMPT_CACHE_FILE=chat.prompt.bin CHAT_SAVE_DIR=./chat/default \ - MODEL=~/llama-models/30B/ggml-model-q4_0. ./examples/chat-persistent.sh +PROMPT_CACHE_FILE=chat.prompt.bin CHAT_SAVE_DIR=./chat/default ./examples/chat-persistent.sh # Resume that chat -PROMPT_CACHE_FILE=chat.prompt.bin CHAT_SAVE_DIR=./chat/default \ - MODEL=~/llama-models/30B/ggml-model-q4_0. ./examples/chat-persistent.sh +PROMPT_CACHE_FILE=chat.prompt.bin CHAT_SAVE_DIR=./chat/default ./examples/chat-persistent.sh # Start a different chat with the same prompt/model -PROMPT_CACHE_FILE=chat.prompt.bin CHAT_SAVE_DIR=./chat/another \ - MODEL=~/llama-models/30B/ggml-model-q4_0. ./examples/chat-persistent.sh +PROMPT_CACHE_FILE=chat.prompt.bin CHAT_SAVE_DIR=./chat/another ./examples/chat-persistent.sh # Different prompt cache for different prompt/model PROMPT_TEMPLATE=./prompts/chat-with-bob.txt PROMPT_CACHE_FILE=bob.prompt.bin \ - CHAT_SAVE_DIR=./chat/bob MODEL=~/llama-models/30B/ggml-model-q4_0. ./examples/chat-persistent.sh + CHAT_SAVE_DIR=./chat/bob ./examples/chat-persistent.sh ``` ### Instruction mode with Alpaca