Broken generate after Add support for batched decoding

[tk-master]

So my script using the generate high level api function got broken after v0.2.13 and I managed to track it down to this commit #795 ab028cb

The model stops responding (eos) and seems to generate random symbols as first character

Here's my test script to demonstrate the issue:

from llama_cpp import Llama

path = '../models/' + 'mistral-7b-instruct-v0.1.Q4_K_M.gguf'
llm = Llama(model_path=path, n_gpu_layers=-1, n_ctx=4096, verbose=True)

history = []

while True:
    user_input = input("\nInput -> ")

    history.append({"role": "user", "content": user_input})

    prompt = "<s>"
    prompt += '[INST] '+ user_input +' [/INST]'
 
    # for msg in history:
    #     if msg['role'] == 'user':
    #         prompt += '[INST] '+ msg['content'] +' [/INST]'
    #     else:
    #         prompt += msg['content'] + '</s>\n'
    print(prompt)

    tok = llm.tokenize(prompt.encode("utf-8"), add_bos=False, special=True)
    print('Prompt tokens: {0}'.format(len(tok)))

    stream = llm.generate(tok, temp=0.3)

    num = 0
    output = ''
    for token in stream:
        num += 1
        if token == llm.token_eos():# or num >= max_tokens:
            print('</s>', end='', flush=True)
            break

        text = llm.detokenize([token])
        text = text.decode("utf-8")
        output += text
        print(text, end='', flush=True)

    #print("\n\nFull response:", output)
    print(f'\nTokens generated: {num}')

    history.append({"role": "assistant", "content": output})

Some example outputs:

Input -> hi
<s>[INST] hi [/INST]
Prompt tokens: 9
↓</s>
Tokens generated: 2


(restarted the script)


Input -> howdy
<s>[INST] howdy [/INST]
Prompt tokens: 10
♂ Hey there! How's life been treating you these days?</s>
Tokens generated: 15

Input -> bruh
<s>[INST] howdy [/INST]♂ Hey there! How's life been treating you these days?</s>
[INST] bruh [/INST]
Prompt tokens: 34
Llama.generate: prefix-match hit
</s>
Tokens generated: 1



(restarted the script)


Input -> hi
<s>[INST] hi [/INST]
Prompt tokens: 9
▲ Hello! How can I help you today?</s>
Tokens generated: 11



(restarted the script)



Input -> howdy
<s>[INST] howdy [/INST]
Prompt tokens: 10
► Hey there! How's life been treating you these days?</s>
Tokens generated: 15

Input -> lol
<s>[INST] lol [/INST]
Prompt tokens: 9
Llama.generate: prefix-match hit
</s>
Tokens generated: 1



(restarted the script)


Input -> hi
<s>[INST] hi [/INST]
Prompt tokens: 9
  Hello! How can I assist you today?</s>
Tokens generated: 11

Input -> how r ya
<s>[INST] hi [/INST]  Hello! How can I assist you today?</s>
[INST] how r ya [/INST]
Prompt tokens: 31
Llama.generate: prefix-match hit
</s>
Tokens generated: 1

Input -> why ..
<s>[INST] hi [/INST]  Hello! How can I assist you today?</s>
[INST] how r ya [/INST]</s>
[INST] why .. [/INST]
Prompt tokens: 42
Llama.generate: prefix-match hit
</s>
Tokens generated: 1

[abetlen]

@tk-master sorry about that, I think it has to do with the kv_cache manipulation functions, let me look into it!

[tk-master]

@tk-master sorry about that, I think it has to do with the kv_cache manipulation functions, let me look into it!

yeah i think it has to do with that cause if i remove this line self._ctx.kv_cache_seq_rm(-1, n_past, -1) it changes nothing, so its not working

[Josh-ee]

I think I am having the same issue:
llama-cpp-python-0.2.11

llama-cpp-python-0.2.15

model_url = 'https://huggingface.co/TheBloke/Mistral-11B-OmniMix-GGUF/resolve/main/mistral-11b-omnimix-    bf16.Q4_K_M.gguf'

llm = LlamaCPP(
        model_url=model_url,
        temperature=0.1,
        max_new_tokens=256,
        context_window=3900,
        generate_kwargs={},
        model_kwargs={'n_gpu_layers': 1},
        verbose=False,
    )

[robertritz]

I can confirm the same issue with multiple models. I thought I broke something in my chat templates.

I was on 0.2.13 and it was fine, just updated to 0.2.15 and noticed this issue.

[abetlen]

@tk-master after some digging I think I found it, can you try setting logits_all=True on the Llama class and seeing if you still get the same issue?

[abetlen]

Almost certain the most recent commit solves the issue.

[gmacgmac]

Hi @abetlen
I am getting empty response on 0.2.15 as well across all GGUF models
this script works and responds with llama-cpp-python v0.2.11 but returns empty responses with v0.2.15
I tried adding "logits_all=True" to "llm = Llama(model_path=filename, n_ctx=512, n_batch=126, n_gpu_layers=1, logits_all=True)" but no difference

import os
import urllib.request
from llama_cpp import Llama

filename = "./dolphin-2.2.1-mistral-7b.Q6_K.gguf"

llm = Llama(model_path=filename, n_ctx=512, n_batch=126, n_gpu_layers=1)

def generate_text(
    prompt="Who is the CEO of Apple?",
    max_tokens=256,
    temperature=0.1,
    top_p=0.5,
    echo=False,
    stop=["#"],
):
    output = llm(
        prompt,
        max_tokens=max_tokens,
        temperature=temperature,
        top_p=top_p,
        echo=echo,
        stop=stop,
    )
    output_text = output["choices"][0]["text"].strip()
    return output_text


def generate_prompt_from_template(input):
    chat_prompt_template = f"""<|im_start|>system
You are a helpful chatbot.<|im_end|>
<|im_start|>user
{input}<|im_end|>"""
    return chat_prompt_template


prompt = generate_prompt_from_template(
    "In 5 words only, tell me what a donkey is."
)

print(generate_text(
    prompt,
    max_tokens=356,
))

def q(query):
    q = generate_prompt_from_template(query)
    print(generate_text(q,max_tokens=356))

-- Logs for v0.2.11 - SUCCESS
`
llama_new_context_with_model: n_ctx = 512
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: kv self size = 64.00 MB
llama_new_context_with_model: compute buffer total size = 23.84 MB
llama_new_context_with_model: max tensor size = 102.55 MB
ggml_metal_add_buffer: allocated 'data ' buffer, size = 5666.81 MB, ( 5667.31 / 73728.00)
ggml_metal_add_buffer: allocated 'kv ' buffer, size = 66.00 MB, ( 5733.31 / 73728.00)
ggml_metal_add_buffer: allocated 'alloc ' buffer, size = 17.97 MB, ( 5751.28 / 73728.00)
AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |

llama_print_timings: load time = 315.10 ms
llama_print_timings: sample time = 15.24 ms / 19 runs ( 0.80 ms per token, 1246.56 tokens per second)
llama_print_timings: prompt eval time = 315.07 ms / 53 tokens ( 5.94 ms per token, 168.22 tokens per second)
llama_print_timings: eval time = 400.54 ms / 18 runs ( 22.25 ms per token, 44.94 tokens per second)
llama_print_timings: total time = 754.20 ms
<|im_start|>system
Domesticated equine mammal.

q("what are you?")

Llama.generate: prefix-match hit

llama_print_timings: load time = 315.10 ms
llama_print_timings: sample time = 52.22 ms / 70 runs ( 0.75 ms per token, 1340.56 tokens per second)
llama_print_timings: prompt eval time = 265.15 ms / 11 tokens ( 24.10 ms per token, 41.49 tokens per second)
llama_print_timings: eval time = 1530.52 ms / 69 runs ( 22.18 ms per token, 45.08 tokens per second)
llama_print_timings: total time = 1926.12 ms
<|im_start|>system
I am an artificial intelligence-based chatbot designed to assist users with various tasks and provide information. I can engage in conversations, answer questions, and perform simple tasks based on the data I have been trained on. My primary purpose is to help and support users in a friendly and efficient manner.
`

-- Logs for v0.2.15 - EMPTY RESPONSE
`
llama_new_context_with_model: n_ctx = 512
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: kv self size = 64.00 MB
llama_build_graph: non-view tensors processed: 740/740
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2 Max
ggml_metal_init: picking default device: Apple M2 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: loading '/Users/gm/miniconda3/envs/textgen2/lib/python3.11/site-packages/llama_cpp/ggml-metal.metal'
ggml_metal_init: GPU name: Apple M2 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple8 (1008)
ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: recommendedMaxWorkingSetSize = 73728.00 MB
ggml_metal_init: maxTransferRate = built-in GPU
llama_new_context_with_model: compute buffer total size = 24.59 MB
llama_new_context_with_model: max tensor size = 102.55 MB
ggml_metal_add_buffer: allocated 'data ' buffer, size = 5666.81 MB, ( 5667.31 / 73728.00)
ggml_metal_add_buffer: allocated 'kv ' buffer, size = 64.02 MB, ( 5731.33 / 73728.00)
ggml_metal_add_buffer: allocated 'alloc ' buffer, size = 17.97 MB, ( 5749.30 / 73728.00)
AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |

llama_print_timings: load time = 260.74 ms
llama_print_timings: sample time = 0.54 ms / 4 runs ( 0.13 ms per token, 7476.64 tokens per second)
llama_print_timings: prompt eval time = 260.70 ms / 31 tokens ( 8.41 ms per token, 118.91 tokens per second)
llama_print_timings: eval time = 68.40 ms / 3 runs ( 22.80 ms per token, 43.86 tokens per second)
llama_print_timings: total time = 337.60 ms
$

q("what are you?")

Llama.generate: prefix-match hit

llama_print_timings: load time = 260.74 ms
llama_print_timings: sample time = 0.15 ms / 1 runs ( 0.15 ms per token, 6802.72 tokens per second)
llama_print_timings: prompt eval time = 273.56 ms / 5 tokens ( 54.71 ms per token, 18.28 tokens per second)
llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: total time = 275.20 ms
`

Thanks for all you do @abetlen :-)

[tk-master]

latest commits fixed it for me, thnx

@gmacgmac try 0.216, should be good now

[gmacgmac]

thanks for alerting @tk-master thought I'm still getting empty responses with v0.2.16

ggml_metal_add_buffer: allocated 'data            ' buffer, size =  5666.81 MB, ( 5667.31 / 73728.00)
ggml_metal_add_buffer: allocated 'kv              ' buffer, size =    64.02 MB, ( 5731.33 / 73728.00)
ggml_metal_add_buffer: allocated 'alloc           ' buffer, size =    17.97 MB, ( 5749.30 / 73728.00)
AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | 

llama_print_timings:        load time =     259.11 ms
llama_print_timings:      sample time =       0.28 ms /     3 runs   (    0.09 ms per token, 10909.09 tokens per second)
llama_print_timings: prompt eval time =     259.03 ms /    31 tokens (    8.36 ms per token,   119.68 tokens per second)
llama_print_timings:        eval time =      45.45 ms /     2 runs   (   22.72 ms per token,    44.01 tokens per second)
llama_print_timings:       total time =     346.90 ms

is this a separate issue to yours then?

[tk-master]

@gmacgmac looks like an issue with your prompt format.. try this

def generate_prompt_from_template(input):
    chat_prompt_template = f"""<|im_start|>system
You are a helpful chatbot.<|im_end|>
<|im_start|>user
{input}<|im_end|>
<|im_start|>assistant"""
    return chat_prompt_template

It just needed <|im_start|>assistant

also I had to change layers to n_gpu_layers=-1 to load all on gpu

[gmacgmac]

can't believe it, 0.2.17 has already been released and it looks like it solved the problem - i think
it's much speedier too
I'll continue testing and advise if there's any issues

amazing, thanks team @abetlen and whomever else supports this project