Misc. bug: Completion fails with error 500

[gvcgael]

Name and Version

build: 5686 (e434e691) with cc (GCC) 15.1.1 20250425 for x86_64-pc-linux-gnu

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server --fim-qwen-1.5b-default

Problem description & steps to reproduce

When using llama with Qwen 1.5b FIM model with llama.vscode, I get, after a couple of minutes, an error 500 on the completion endpoint.

First Bad Commit

it seems to be happening around :

3555b3004ba7687be3d734acade52a3345758aa4 is the first bad commit
commit 3555b3004ba7687be3d734acade52a3345758aa4 (HEAD, tag: b5675)
Author: xctan <xc-tan@outlook.com>
Date:   Mon Jun 16 13:54:15 2025 +0800

    ggml-cpu : rework weak alias on apple targets (#14146)
    
    * ggml-cpu : rework weak alias on apple targets
    
    * fix powerpc detection
    
    * fix ppc detection
    
    * fix powerpc detection on darwin

 ggml/cmake/common.cmake            |  3 +-
 ggml/src/ggml-cpu/apple-fallback.h | 88 +++++++++++++++++++++++++++++++++++++
 ggml/src/ggml-cpu/ggml-cpu-impl.h  |  2 +-
 ggml/src/ggml-cpu/quants.c         |  4 ++
 ggml/src/ggml-cpu/quants.h         | 27 ------------
 ggml/src/ggml-cpu/repack.cpp       |  4 ++
 ggml/src/ggml-cpu/repack.h         | 18 +-------
 7 files changed, 99 insertions(+), 47 deletions(-)

If a checkout the master branch state for a day earlier it seems to work fine

Relevant log output

decode: failed to initialize batch
llama_decode: failed to decode, ret = -1
srv  update_slots: Invalid input batch., i = 0, n_batch = 1024, ret = -1
slot      release: id  0 | task 2529 | stop processing: n_past = 5594, truncated = 0
srv    send_error: task id = 2529, error: Invalid input batch.
srv  update_slots: all slots are idle
srv  cancel_tasks: cancel task, id_task = 2529
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /infill 127.0.0.1 500
slot launch_slot_: id  0 | task 2532 | processing task
slot update_slots: id  0 | task 2532 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 5594
slot update_slots: id  0 | task 2532 | need to evaluate at least 1 token for each active slot, n_past = 5594, n_prompt_tokens = 5594
slot update_slots: id  0 | task 2532 | kv cache rm [5593, end)
slot update_slots: id  0 | task 2532 | prompt processing progress, n_past = 5594, n_tokens = 1, progress = 0.000179
slot update_slots: id  0 | task 2532 | prompt done, n_past = 5594, n_tokens = 1
init: sequence 0 does not start from the last position stored in the memory
decode: failed to initialize batch
llama_decode: failed to decode, ret = -1

[rotemdan]

I'm experiencing the same issue after building from latest repository, and using the C library directly. (OS tested so far: Windows 11 with CPU and CUDA builds).

The issue happens with any model I tried.

Based on my experimentation, it happens when a completion is aborted abruptly but the next time the context is reused, llama_kv_self_seq_rm is not implicitly called, meaning that calls to llama_decode may be at a token position where there are existing token entries in the sequence beyond the position where llama_decode is told to start.

This issue never happened with builds from a few days ago. Everything worked without error.

If I change my code to unconditionally call llama_kv_self_seq_rm starting at the token position where decoding should start, effectively truncating the sequence, before calling llama_decode, it seems to prevent this error.

It seems to either be a bug or an undocumented breaking change in behavior - I'm not sure if it's intentional or not.

[rotemdan]

May be related to #14217. It involved many changes to llama-batch.cpp and related source files.

[ggerganov]

@gvcgael Could you check if #14339 fixes the issue for you?

@rotemdan This might still not fix you problem. After #14217, llama_decode() now requires that the new batch starts from the end of what is currently stored in the context. There wasn't a check for this before, so it is possible that you always had this problem, but it remained hidden.

Based on my experimentation, it happens when a completion is aborted abruptly but the next time the context is reused, llama_kv_self_seq_rm is not implicitly called

When a llama_decode() is aborted, the successful ubatches before the abort remain in the context. We only remove the unsuccessful ubatch:

llama.cpp/src/llama-context.cpp

Lines 1014 to 1045 in 3ddb77b

	if (!res) {
	// the last ubatch failed or was aborted -> remove all positions of that ubatch from the KV cache
	llama_pos pos_min[LLAMA_MAX_SEQ];
	for (int s = 0; s < LLAMA_MAX_SEQ; ++s) {
	pos_min[s] = std::numeric_limits<llama_pos>::max();
	}
	// TODO: fix sequence indexing
	for (uint32_t i = 0; i < ubatch.n_tokens; ++i) {
	const auto & seq_id = ubatch.seq_id[i][0];
	pos_min[seq_id] = std::min(pos_min[seq_id], ubatch.pos[i]);
	}
	for (int s = 0; s < LLAMA_MAX_SEQ; ++s) {
	if (pos_min[s] == std::numeric_limits<llama_pos>::max()) {
	continue;
	}
	LLAMA_LOG_WARN("%s: removing KV cache entries for seq_id = %d, pos = [%d, +inf)\n", __func__, s, pos_min[s]);
	memory->seq_rm(s, pos_min[s], -1);
	}
	switch (status) {
	case GGML_STATUS_ABORTED: return 2;
	case GGML_STATUS_ALLOC_FAILED: return -2;
	case GGML_STATUS_FAILED: return -3;
	case GGML_STATUS_SUCCESS: GGML_ABORT("should not happen");
	}
	}

To handle such cases, you have to query the context about the maximum position for each sequence from the batch via llama_memory_seq_pos_max() and update your application state to discard any tokens that could incorrectly be assumed as processed by the aborted llama_decode().

[rotemdan]

When I wrote "abruptly aborted" I guess I meant that decoding is aborted in the sense that there is no tracking, in my code, on the state of the sequence, and then when context is reused I didn't always ensure to truncate before calling llama_decode. I did call llama_kv_self_seq_rm in some cases, for partial context reuse, but not consistently. Once I changed to call it consistently I'm not seeing the error anymore so far.

I don't actually terminate the thread that runs llama_decode (that would be likely unsafe), but I understand that it's better the library should be robust towards that.

I just noticed that the kv_ sequence methods became deprecated. I'll switch to use the llama_memory_ methods instead (seem to require llama_memory_t - need to figure out how that works).

Anyway, I'm not sure what is exactly is/was causing llama-server to fail. I have personally never actually used it myself, so that isn't really my area of the issue report.

[rotemdan]

Alright, I updated my code to use the llama_memory_ methods, and ensured that if I don't call llama_memory_seq_rm, before llama_decode, in some cases (context reuse) I get the same error:

sequence 0 does not start from the last position stored in the memory

The thing is the error seems to be confusing (the new pull request #14339 didn't change the error message significantly).

If the issue is truly that the earliest token position in the given batch is smaller or equal to the last token position in the memory (KV cache), meaning it intersects the current sequence, I would expect it to report that directly.

But instead the message seems to imply that the sequence 0 itself, doesn't start from the last position (what does that mean?), which seems, based on the way it's phrased, to be unrelated or nonsensical.

[ggerganov]

@rotemdan Thanks for the feedback. I just pushed an alternative wording for the error: 195c1e3

Is this clearer? Any additional suggestions how to improve it are welcome.

[rotemdan]

Thanks a lot. Highly appreciated. That would help to more easily debug other instances of the issue or similar ones, for other people or projects.

I gave the code for llama_batch_allocr::init to Gemini 2.5 Flash and asked it to summarize all the validation checks being done.

It replied with:

---

1. Batch Size: The batch must contain at least one token.
2. Token Validity:
   • If token IDs are provided, each token ID must be non-negative.
   • Each token ID must be within the valid range of the vocabulary (i.e., less than the total number of tokens in the vocabulary).
3. Sequence ID Validity:
   • If sequence IDs are provided, each sequence ID must be non-negative.
   • Each sequence ID must be less than the maximum allowed sequence ID (LLAMA_MAX_SEQ).
4. Positional Continuity (with Memory):
   • If a memory state is provided:
     • For batches with tokenized input, the starting position of any sequence in the batch must be exactly one greater than its last recorded position in the memory.
     • For batches with embedding input, the starting position of any sequence in the batch must be either the same as or one greater than its last recorded position in the memory.
5. Positional Continuity (within Batch):
   • For any given sequence, the positions assigned to its tokens within the batch must be continuous (i.e., no gaps between the minimum and maximum positions covered by that sequence in the batch).
   • Positions for any given sequence must be non-decreasing throughout the batch.
6. Coupled Sequence Consistency (with Memory):
   • If a memory state is provided, and two sequences are "coupled" (meaning they share at least one token in the current batch), then their historical minimum and maximum positions in memory must be identical. This ensures that coupled sequences have not diverged in their past processing history.
7. Sequence Set Compatibility (within Batch):
   • For any given sequence, the set of other sequences it is coupled with (i.e., shares tokens with) must remain compatible as the batch progresses. This means a sequence cannot suddenly belong to a subset of coupled sequences that is incompatible with its previous associations within the same batch.

---

This information can be helpful to anyone encountering similar issues with llama_decode. I can't, at the moment, suggest any additional checks, but I could try to see if I can suggest modifications to the existing error messages.

[gvcgael]

@gvcgael Could you check if #14339 fixes the issue for you?

I used this branch for a couple of hours and it seems to have fixed this issue. Thanks for the quick fix