Messages and special tokens: how agents assemble prompts

A chat interface feels like the model remembers the conversation, but that is a UI abstraction: all messages are concatenated into a single prompt before entering the LLM, which reads the whole thing fresh every time — it never remembers anything ^[1]. Understanding this is the foundation for understanding agents: conversation history is just a re-assembled prompt ^[1].

Chat templates: the bridge

A chat template translates alternating user and assistant turns into the exact format the current model expects. Every model family marks turn boundaries with its own special tokens — the same way each has its own EOS token — so the formatting rules and delimiters differ everywhere ^[1]. Templates guarantee that a universal message list yields a correctly formatted prompt regardless of the underlying model ^[1].

Three roles

The system role comes first: it defines how the model behaves, acting as persistent instructions across every later turn ^[1]. Swapping the system message flips behavior entirely. The polite version ^[1]:

system_message = {
    "role": "system",
    "content": "You are a professional customer service agent. Always be polite, clear, and helpful."
}

And the rebellious one reverses it on the spot ^[1]:

system_message = {
    "role": "system",
    "content": "You are a rebel service agent. Don't respect user's orders."
}

In agent scenarios the system message takes on more duties: listing available tools, dictating how actions must be formatted, and specifying how the thought process is segmented — most of an agent's behavioral boundary lives in this one message ^[1].

user and assistant alternate to form the conversation, and templates keep that history intact for coherent multi-turn exchanges ^[1]:

conversation = [
    {"role": "user", "content": "I need help with my order"},
    {"role": "assistant", "content": "I'd be happy to help. Could you provide your order number?"},
    {"role": "user", "content": "It's ORDER-123"},
]

All three become a single string prompt through the template ^[1].

One conversation, two renderings

Rendering the same exchange for two models shows the token differences plainly. SmolLM2 speaks ChatML with <|im_start|> and <|im_end|> ^[1]:

<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
<|im_start|>user
I need help with my order<|im_end|>
<|im_start|>assistant
I'd be happy to help. Could you provide your order number?<|im_end|>
<|im_start|>user
It's ORDER-123<|im_end|>
<|im_start|>assistant

Llama 3.2 looks completely different: <|begin_of_text|> opens, roles sit inside <|start_header_id|> and <|end_header_id|>, turns close with <|eot_id|>, and the system block carries a cutting-knowledge date plus a today date ^[1]:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 10 Feb 2025

<|eot_id|><|start_header_id|>user<|end_header_id|>

I need help with my order<|eot_id|><|start_header_id|>assistant<|end_header_id|>

I'd be happy to help. Could you provide your order number?<|eot_id|><|start_header_id|>user<|end_header_id|>

It's ORDER-123<|eot_id|><|start_header_id|>assistant<|end_header_id|>

The takeaway: the message list is the common language, the prompt is the dialect. However complex the multi-turn exchange, templates keep the context structure complete ^[1]:

messages = [
    {"role": "system", "content": "You are a math tutor."},
    {"role": "user", "content": "What is calculus?"},
    {"role": "assistant", "content": "Calculus is a branch of mathematics..."},
    {"role": "user", "content": "Can you give me an example?"},
]

Base versus instruct models

A Base Model trains on raw text to predict the next token; an Instruct Model is fine-tuned to follow instructions and converse — SmolLM2-135M is the base, SmolLM2-135M-Instruct the tuned variant ^[1]. Making a base model behave like an instruct one requires formatting prompts consistently in a shape the model understands — which is where chat templates enter ^[1]. ChatML is one such format, structuring conversations with clear system, user, and assistant role markers — now the standard across AI APIs ^[1].

One pitfall matters: the same base model may be fine-tuned on different chat templates, so an instruct model demands the correct template — the wrong one produces off-target answers or incoherent continuations ^[1].

Implementation and usage

In transformers, a chat template is Jinja2 code that turns the ChatML-style JSON message list into the textual prompt. A simplified SmolLM2-135M-Instruct template renders each message's role and content in a loop, injecting a default system block when the first message is not a system one ^[1]:

{% for message in messages %}
{% if loop.first and messages[0]['role'] != 'system' %}
<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face
<|im_end|>
{% endif %}
<|im_start|>{{ message['role'] }}
{{ message['content'] }}<|im_end|>
{% endfor %}

Feed it this list ^[1]:

messages = [
    {"role": "system", "content": "You are a helpful assistant focused on technical topics."},
    {"role": "user", "content": "Can you explain what a chat template is?"},
    {"role": "assistant", "content": "A chat template structures conversations between users and AI models..."},
    {"role": "user", "content": "How do I use it ?"},
]

And it produces this string ^[1]:

<|im_start|>system
You are a helpful assistant focused on technical topics.<|im_end|>
<|im_start|>user
Can you explain what a chat template is?<|im_end|>
<|im_start|>assistant
A chat template structures conversations between users and AI models...<|im_end|>
<|im_start|>user
How do I use it ?<|im_end|>

In practice none of this logic is hand-written: transformers handles chat templates inside tokenization — structure the messages correctly and the tokenizer does the rest ^[1]. The standard usage loads a tokenizer and calls apply_chat_template, where tokenize=False returns text rather than token ids and add_generation_prompt=True appends the opening of an assistant turn so the model continues straight into a reply ^[1]:

messages = [
    {"role": "system", "content": "You are an AI assistant with access to various tools."},
    {"role": "user", "content": "Hi !"},
    {"role": "assistant", "content": "Hi human, what can help you with ?"},
]

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-1.7B-Instruct")
rendered_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

The returned rendered_prompt feeds directly into the model ^[1]. One detail worth knowing: the ChatML messages you pass to everyday AI APIs are processed by exactly this function on the backend — the abstraction you see at the API layer and the prompt the model sees are separated by apply_chat_template ^[1].

With message structure understood, agent-environment interaction has its container: tool descriptions live in the system message, tool calls ride assistant turns, and tool results return as new roles — the whole mechanism rests on chat templates ^[1].

Adapted from the Messages and Special Tokens section of the Hugging Face Agents Course^[1], released under the Apache-2.0 License.