0.2.9 broke the bos/eos/sys handling for chat sequences

[earonesty]

recent changes (0.2.7 -> 0.2.9.) to the chat framework broke things (this is vicuna 7b)

first call always returns the prompt:

[INST] <<SYS>> You are a helpful assistant. <|endoftext|>

second call works:

Once upon a time in a faraway land, there lived a silly frog named Freddy who loved to hop around on lily pads and croak loudly.

third and continuing calls get odder, and show the bos tokens, etc:

[INST] <<SYS>>
You are a helpful assistant.
<</SYS>>



The silly frog hops around on the lily pad, croaking loudly.

trying different versions.

1. 0.2.7 fixes it
2. 0.2.9 breaks it
3. 0.2.11 still broken

its a token handling issue. gonna try to figure out what broke between 0.2.7 and 0.2.9

looks like it was this:

"+- Add configurable chat formats by @abetlen in #711"

[earonesty]

verified that this PR does not fully fix the issue, leaving most users locked to version 0.2.7

#808

[earonesty]

easy to reproduce, just specify a longer chat conversation. at least 3 user and 2 assistant messages result in weird results

[abetlen]

@earonesty just confirming that #850 didn't resolve this?

[earonesty]

correct I tried that branch and while it fixes the first request it does not fix subsequent requests

It's really about subsequent requests with a persistent running instance

[earonesty]

I suspect something about the vectors not getting cleared between requests or something is going on

[earonesty]

tried again today with v 0.2.13:

$ python example.py
This is an instruction to write a sentence about a silly frog in the next message.
(venv)
erik@LAPTOP-1K8B9NQ8 MINGW64 ~/chainer (main)
$ python example.py
You have been asked to write a description of an absurd scenario, and the only mental image you can conjure is that of a silly-looking frog. It jumps around in your mind, its skin green and
bumpy like a piece of broccoli, and it hops along with a goofy grin on its face. You wonder how such a creature would behave if it were to come to life. Maybe it would croak out a silly joke
 or do a funny dance. Or perhaps it would just sit there, looking completely bewildered by the world around it.
(venv)
erik@LAPTOP-1K8B9NQ8 MINGW64 ~/chainer (main)
$ python example.py
[INST]Write one paragraph about why the frog is silly. [/INST]

you can see that it starts hallucinating system messages and prompts, instead of generating responses to the "write one sentence about a silly frog". this is vicuna. version 0.2.7 just works every time.

[earonesty]

hmm, looks like maybe this is a problem a little deeper. i can see the engine is formatting the input correctly, and each request looks normal

<<SYS>>
You are a helpful assistant.
<</SYS>>



[INST]Write one sentence about a silly frog. [/INST]

it's only in subsequent requests that it starts getting odd. tried disabling caching, didn't help. eventually it starts spitting out system prompts as completions.

it's like it's chaining my requests, instead of starting over.

look at the output of the 3rd request.... two prompts. and each is progressive.

[INST]Write another sentence that is completely unrelated to the first sentence. [/INST]
[INST]Write a third sentence that connects the two sentences in a strange way. [/INST]

[earonesty]

yes, that PR and the other fixed it.

you now need to specify the chat_format correctly ... it won't guess anymore!