Building Agentic RAG Systems with smolagents

Traditional RAG is a straight line: the user query goes to a retriever, the fetched snippets are stuffed into the model together with the question, and the model answers. The line has two built-in weaknesses — there is exactly one retrieval chance, so whatever the first round misses is gone for good; and it aligns on literal semantic similarity, so a slightly off phrasing can push genuinely relevant material outside the recall window ^[1].

Agentic RAG hands the retrieval process to an agent: it formulates search queries autonomously, critiques what comes back, and runs as many retrieval rounds as needed. Retrieval stops being a fixed pipeline stage and becomes a set of actions the model can plan and iterate on ^[1].

The simplest version: a web-search agent

A web-connected retrieval agent in smolagents needs one search tool and one model:

from smolagents import CodeAgent, DuckDuckGoSearchTool, InferenceClientModel

# Initialize the search tool
search_tool = DuckDuckGoSearchTool()

# Initialize the model
model = InferenceClientModel()

agent = CodeAgent(
    model=model,
    tools=[search_tool],
)

# Example usage
response = agent.run(
    "Search for luxury superhero-themed party ideas, including decorations, entertainment, and catering."
)
print(response)

The agent runs in four steps: analyze the request and identify the key elements of the query; perform retrieval with DuckDuckGo for the most relevant and up-to-date information; synthesize the scattered results into a cohesive, actionable plan; and store what was retrieved so future planning tasks can reuse it ^[1].

Custom knowledge base: wrap the retriever as a tool

Web search covers open information, but enterprise settings more often mean turning private documents into a knowledge base. The recipe is to wrap the retriever as a smolagents Tool: a vector database stores numerical representations (embeddings) of text created by machine learning models, and enables semantic search by identifying similar meanings in high-dimensional space ^[1]. The example pairs a BM25 retriever with RecursiveCharacterTextSplitter:

from langchain_community.docstore.document import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from smolagents import Tool
from langchain_community.retrievers import BM25Retriever
from smolagents import CodeAgent, InferenceClientModel

class PartyPlanningRetrieverTool(Tool):
    name = "party_planning_retriever"
    description = "Uses semantic search to retrieve relevant party planning ideas for Alfred’s superhero-themed party at Wayne Manor."
    inputs = {
        "query": {
            "type": "string",
            "description": "The query to perform. This should be a query related to party planning or superhero themes.",
        }
    }
    output_type = "string"

    def __init__(self, docs, **kwargs):
        super().__init__(**kwargs)
        self.retriever = BM25Retriever.from_documents(
            docs, k=5  # Retrieve the top 5 documents
        )

    def forward(self, query: str) -> str:
        assert isinstance(query, str), "Your search query must be a string"

        docs = self.retriever.invoke(
            query,
        )
        return "\nRetrieved ideas:\n" + "".join(
            [
                f"\n\n===== Idea {str(i)} =====\n" + doc.page_content
                for i, doc in enumerate(docs)
            ]
        )

The tool class is a textbook smolagents pattern: name and description are what the model reads to decide when to call the tool, inputs declares the argument schema, and forward does the actual work — pulling the top 5 documents from the BM25 index and formatting them with separators. Then the corpus is prepared, chunked, and wired into an agent:

party_ideas = [
    {"text": "A superhero-themed masquerade ball with luxury decor, including gold accents and velvet curtains.", "source": "Party Ideas 1"},
    {"text": "Hire a professional DJ who can play themed music for superheroes like Batman and Wonder Woman.", "source": "Entertainment Ideas"},
    {"text": "For catering, serve dishes named after superheroes, like 'The Hulk's Green Smoothie' and 'Iron Man's Power Steak.'", "source": "Catering Ideas"},
    {"text": "Decorate with iconic superhero logos and projections of Gotham and other superhero cities around the venue.", "source": "Decoration Ideas"},
    {"text": "Interactive experiences with VR where guests can engage in superhero simulations or compete in themed games.", "source": "Entertainment Ideas"}
]

source_docs = [
    Document(page_content=doc["text"], metadata={"source": doc["source"]})
    for doc in party_ideas
]

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    add_start_index=True,
    strip_whitespace=True,
    separators=["\n\n", "\n", ".", " ", ""],
)
docs_processed = text_splitter.split_documents(source_docs)

# Create the retriever tool
party_planning_retriever = PartyPlanningRetrieverTool(docs_processed)

# Initialize the agent
agent = CodeAgent(tools=[party_planning_retriever], model=InferenceClientModel())

# Example usage
response = agent.run(
    "Find ideas for a luxury superhero-themed party, including entertainment, catering, and decoration options."
)

print(response)

The chunking parameters deserve attention: chunk_size 500 with chunk_overlap 50, start indexes added, whitespace stripped, and separators going all the way down from paragraph level to single characters. As a rule of thumb, oversized chunks dilute retrieval precision, undersized chunks lose context, and an overlap of around 50 characters keeps sentences that straddle chunk boundaries intact. The assembled agent first checks the documentation for relevant information, combines insights from the knowledge base, and maintains conversation context in memory ^[1].

BM25 and dense retrieval complement each other: BM25 is a term-statistics method that needs no embedding model and is strong at exact terminology matches, while dense retrieval captures paraphrases and semantic near-neighbors. Production systems often merge both recall paths as hybrid search and rerank the combined list.

Making retrieval smarter

The advanced playbook for agentic RAG clusters into seven strategies: query reformulation, crafting search terms that better match target documents instead of reusing the raw query; query decomposition, splitting compound questions into multiple sub-queries; query expansion, rephrasing one intent in several wordings and searching them all; reranking, scoring query-document pairs with a Cross-Encoder for finer-grained relevance; multi-step retrieval, letting initial results inform subsequent queries; source integration, combining web search with local documentation; and result validation, screening retrieved content for relevance and accuracy before it enters the answer ^[1].

Each strategy has its engineering niche. Cross-Encoder scoring is accurate but expensive — every pair runs a full model pass — so the standard pattern is a cheap first-stage recall of top-k candidates followed by a rerank of a much shorter list. Query decomposition fits compound questions with multiple entities or constraints, retrieving sub-queries independently and merging with deduplication. Result validation can ask the model to judge factual consistency between snippets and the question; inconsistent snippets are better left unused.

System-level design matters just as much: the agent should pick between available tools based on query type and context; memory systems maintain conversation history and avoid repetitive retrievals; fallback strategies keep the system useful when primary retrieval fails; and validation steps hold the final quality gate ^[1].

Adapted from the "Building Agentic RAG Systems" section of Hugging Face's agents-course^[1], released under the Apache-2.0 License.

^[1]: Building Agentic RAG Systems | Hugging Face Agents Course