How to use the assistant_prefill with the thinking mode

[abdullah-cod9]

I am applying the following example:

messages = [ {"role": "user", "content": "What are the first 5 planets in the solar system?"}, {"role": "assistant", "content": "...Wait, I've thought long enough, let's answer.</think>"} ]

# Seamlessly continue the generation
response = visionM.model.create_chat_completion( messages=messages, max_tokens=512, assistant_prefill=True # <--- Enables seamless continuation )

prefilled_text = messages[-1]["content"]
# The model will flawlessly continue from " Venus\n3. Earth..."
generated_text = response["choices"][0]["message"]["content"]

print(prefilled_text + generated_text)

The model rethinks and does not jump to a answer.

Chat Handler:

Qwen35ChatHandler(_init_mtmd_context): Vision support detected. Qwen35ChatHandler(_init_mtmd_context): Audio is NOT supported by this mmproj model backend. Qwen35ChatHandler(_process_mtmd_prompt): Rendered prompt length: 658 chars, Media count: 0. Qwen35ChatHandler(_process_mtmd_prompt): Rendered prompt: <|im_start|>system You are an exceptionally capable, precise, and helpful multimodal AI assistant that excels at deeply understanding and richly describing images, charts, diagrams, text in images, scenes, and any visual content, while also answering every question accurately, clearly, and step-by-step when appropriate — always responding in the same language as the user's question, remaining polite, professional, and maximally helpful.<|im_end|> <|im_start|>user
What are the first 5 planets in the solar system?<|im_end|> <|im_start|>assistant <think> ...Wait, I've thought long enough, let's answer. </think>

<|im_end|> <|im_start|>assistant <think>

Qwen35ChatHandler(__call__): Prepared virtual token ledger of length 130. Llama.longest_token_prefix [Fast Exit 1]: Empty sequence detected. len(current_ids)=0, len(new_tokens)=130 Qwen35ChatHandler(__call__): Evaluating TEXT chunk (130 tokens) at pos 0... ...Wait, I've thought long enough, let's answer.</think>Thinking Process:

1. **Analyze the Request:** * Question: "What are the first 5 planets in the solar system?" * Constraint: Respond in the same language as the user's question (English). * Constraint: Be polite, professional, and maximally helpful. * Constraint: Step-by-step reasoning when appropriate (though this is a factual question, I should explain the order clearly).

[JamePeng]

set the create_chat_completion(add_generation_prompt=False) when you use the assistant_prefill

[abdullah-cod9]

Thanks for the reply.

Setting add_generation_prompt=False does stop the model from starting a new assistant turn, but in my case it introduces another issue
the model frequently jumps into tool calling instead of continuing the assistant message.

---

Small update: after further testing, it looks like I misinterpreted the behavior.

With add_generation_prompt=False, the model is not actually jumping into tool calling. Instead, no continuation is generated at all. response["choices"][0]["message"]["content"] is empty, so the final output remains only the prefilled text.

Example:

messages = [
    {"role": "user", "content": "What are the first 5 planets in the solar system?"},
    {
        "role": "assistant",
        "content": "...Wait, I've thought long enough, let's answer.</think>",
    },
]

response = visionM.model.create_chat_completion(
    messages=messages,
    max_tokens=1024,
    assistant_prefill=True,
    add_generation_prompt=False,
)

print(repr(response["choices"][0]["message"]["content"]))
# ''

The result is an empty string, so generation appears to stop immediately after the prefilled assistant content rather than continuing from it. Is there something else required for assistant_prefill to work correctly with Qwen 3.5?

[JamePeng]

I didn't quite understand your approach. Shouldn't the continuation writing function send back the unfinished text previously written by the AI? Why give it continuation writing capability on the first try? If you're just trying to control the thought process to prevent it from becoming too long, you can directly use the latest Reasoning Budget.

[abdullah-cod9]

My goal was to prevent the model’s reasoning from becoming too long.

I didn’t assume that llama-cpp-python supports a proper “reasoning budget,” so I tried assistant_prefill as a workaround, inspired by Thread-Safe Abort & Assistant Prefill Integration.

Thanks for the help — I’ll make sure to read the documentation more carefully next time.