[BugFix] Fix Qwen parsing for tools

[vmoens]

Fixes Qwen parsing for tools:

from torchrl.data.llm.history import History
from tensordict import set_list_to_stack
from transformers import AutoTokenizer

set_list_to_stack(True).set()

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

h = History(
    role=["system", "user", "assistant", "tool"],
    content=["You are a helpful assistant. Answer in <answer>...</answer> blocks, think in <think>...</think> blocks.", 
             "Write a python script that gives the capital of France or Germany.", 
             """<think>The capital of France is Paris, the capital of Germany is Berlin.</think> <answer><python>
capitals = {
    "France": "Paris",
    "Germany": "Berlin"
}
def get_capital(country):
    return capitals.get(country, "Unknown")
</python></answer>""", ""],
    tool_calls=[None, None, None, [
    {
        "id": "call_123",  # Unique identifier for this call
        "type": "function",  # Usually "function"
        "function": {
            "name": "get_capital",
            "arguments": '{"country": "Germany"}'  # JSON string
        }
    }
]],
    batch_size=4, tool_responses=[None, None, None, "Berlin"]
)

parsed = h.apply_chat_template(tokenizer=tokenizer, add_generation_prompt=False)
print(parsed)

produces

<|im_start|>system
You are a helpful assistant. Answer in <answer>...</answer> blocks, think in <think>...</think> blocks.<|im_end|>
<|im_start|>user
Write a python script that gives the capital of France or Germany.<|im_end|>
<|im_start|>assistant
<think>The capital of France is Paris, the capital of Germany is Berlin.</think> <answer><python>
capitals = {
    "France": "Paris",
    "Germany": "Berlin"
}
def get_capital(country):
    return capitals.get(country, "Unknown")
</python></answer><|im_end|>
    <|im_start|>tool
<tool_response>
Berlin
</tool_response><|im_end|>

Notice

• the role is now tool
• the tool response is printed

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3087

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• linux.aws.h100.8 instance is down, potentially longer queue on linux.aws.h100

❌ 3 New Failures, 6 Unrelated Failures

As of commit f282268 with merge base b6d2156 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  RuntimeError: PassManager::run failed
• Continuous Benchmark / GPU Pytest benchmark (gh)
  RuntimeError: PassManager::run failed
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t b128170ba976a6146a5c3b47c05774a8120cba7d5975fce7e48e2452cac37024 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Continuous Benchmark / CPU Pytest benchmark (gh) (trunk failure)
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh) (trunk failure)
  AttributeError: module 'torch.utils._pytree' has no attribute 'register_pytree_node'
• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh) (trunk failure)
  test/test_distributed.py::TestRayCollector::test_ray_collector_policy_constructor
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh) (trunk failure)
  AttributeError: module 'torch.utils._pytree' has no attribute 'register_pytree_node'
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh) (trunk failure)
  test/test_distributed.py::TestRayCollector::test_ray_collector_policy_constructor

This comment was automatically generated by Dr. CI and updates every 15 minutes.