LlamaIndex core components: building a RAG pipeline

This guide is for readers who can write a little Python and want a model to answer questions from their own documents. By the end you will know the five stages of a RAG workflow, how to load, split, embed, store and query data with a few lines of code, and how to score the answers.

What LlamaIndex components are for

An LLM learns general knowledge from its training data. It may not have seen your documents, and it may not know the latest information. Retrieval-Augmented Generation (RAG) solves this by finding relevant information in your own data and passing it to the model. LlamaIndex splits this workflow into components. This guide focuses on the QueryEngine, because it can be used as a RAG tool for an agent^[1].

A RAG workflow has five key stages: loading, indexing, storing, querying and evaluation^[1].

Step 1: Load your documents

Loading means bringing data from where it lives into the workflow. That can be text files, PDFs, another website, a database or an API^[1]. LlamaIndex offers three main ways to load data:

• SimpleDirectoryReader, a built-in loader for various file types in a local directory;
• LlamaParse, LlamaIndex's official PDF parsing tool, available as a managed API;
• LlamaHub, a registry of hundreds of data-loading libraries^[1].

The simplest starting point is SimpleDirectoryReader, which converts files into Document objects^[1]:

from llama_index.core import SimpleDirectoryReader

reader = SimpleDirectoryReader(input_dir="path/to/directory")
documents = reader.load_data()

Step 2: Split and embed

A Document is usually too long to search directly, so it is broken into smaller pieces called Node objects. A Node is a chunk of text from the document that still keeps a reference to its Document^[1].

An IngestionPipeline creates nodes with two transformations. SentenceSplitter splits text at natural sentence boundaries, and HuggingFaceEmbedding converts each chunk into an embedding, a numerical vector that represents its meaning^[1]. The example below uses BAAI/bge-small-en-v1.5 as the embedding model, and chunk_overlap=0 means neighbouring chunks do not overlap^[1]:

from llama_index.core import Document
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.ingestion import IngestionPipeline

# create the pipeline with transformations
pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_overlap=0),
        HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5"),
    ]
)

nodes = await pipeline.arun(documents=[Document.example()])

Step 3: Store the vectors and build an index

The vectors need a place to live. The example uses Chroma, so first install its LlamaIndex integration^[1]:

pip install llama-index-vector-stores-chroma

Next, connect the pipeline to the vector store. Setting chunk_size=25 makes the chunks shorter. When the pipeline runs, the vectors are written straight into Chroma and saved in the directory you give it^[1]:

import chromadb
from llama_index.vector_stores.chroma import ChromaVectorStore

db = chromadb.PersistentClient(path="./alfred_chroma_db")
chroma_collection = db.get_or_create_collection("alfred")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=25, chunk_overlap=0),
        HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5"),
    ],
    vector_store=vector_store,
)

The index lets queries find relevant chunks. VectorStoreIndex places the query and the chunks in the same vector space and matches them by similarity. Use the same embedding model that was used during ingestion, so the vectors can be compared consistently^[1]:

from llama_index.core import VectorStoreIndex
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
index = VectorStoreIndex.from_vector_store(vector_store, embed_model=embed_model)

Step 4: Query the index

For any indexing strategy there are many ways to query: sub-queries, multi-step queries and hybrid strategies, all of which can combine LLMs with LlamaIndex data structures^[1]. Indexing can also use metadata strategies, which find contextually relevant data based on the properties of the documents^[1].

An index can be converted into one of three query interfaces^[1]:

• as_retriever returns a list of NodeWithScore objects with similarity scores, so you get the matching chunks only;
• as_query_engine returns a written answer to a single question, and is the more common choice for agent-style interactions;
• as_chat_engine keeps the chat history across multiple messages.

as_query_engine needs an LLM to write the answer. The example below uses HuggingFaceInferenceAPI with Qwen/Qwen2.5-Coder-32B-Instruct and sets response_mode to tree_summarize^[1]:

from llama_index.llms.huggingface_api import HuggingFaceInferenceAPI

llm = HuggingFaceInferenceAPI(model_name="Qwen/Qwen2.5-Coder-32B-Instruct")
query_engine = index.as_query_engine(
    llm=llm,
    response_mode="tree_summarize",
)
query_engine.query("What is the meaning of life?")
# The meaning of life is 42

A ResponseSynthesizer decides how the answer is assembled. The three common strategies are^[1]:

• refine, which goes through each retrieved chunk in turn and refines the answer, with one LLM call per chunk;
• compact, the default, which concatenates the chunks first and makes fewer LLM calls;
• tree_summarize, which builds the answer from each chunk and organises it as a tree.

If you need control over every step of a query, the low-level composition API lets you customise and fine-tune the process. It also works well with Workflows^[1].

Step 5: Evaluate and observe

A model will not always answer correctly, so the answers need scoring. LlamaIndex has three built-in evaluators that use an LLM to judge a response^[1]:

• FaithfulnessEvaluator checks whether the answer is supported by the context;
• AnswerRelevancyEvaluator checks whether the answer is relevant to the question;
• CorrectnessEvaluator checks whether the answer is correct.

The code below uses FaithfulnessEvaluator to check one query. The result is in eval_result.passing^[1]:

from llama_index.core.evaluation import FaithfulnessEvaluator

query_engine = # from the previous section
llm = # from the previous section

# query index
evaluator = FaithfulnessEvaluator(llm=llm)
response = query_engine.query(
    "What battles took place in New York City in the American Revolution?"
)
eval_result = evaluator.evaluate_response(response=response)
eval_result.passing

As workflows grow more complex, single evaluations are not enough, and you also need to see how each component performs. The LlamaTrace callback, which is based on Arize Phoenix, provides that view. Install it with^[1]:

pip install -U llama-index-callbacks-arize-phoenix

Before using it, create an account at LlamaTrace, generate an API key in your account settings, and set the PHOENIX_API_KEY environment variable to that key^[1]. The code that turns on tracing is:

import llama_index
import os

PHOENIX_API_KEY = "<PHOENIX_API_KEY>"
os.environ["OTEL_EXPORTER_OTLP_HEADERS"] = f"api_key={PHOENIX_API_KEY}"
llama_index.core.set_global_handler(
    "arize_phoenix",
    endpoint="https://llamatrace.com/v1/traces"
)

Next steps

Put these components together and you have a working RAG pipeline. The next step is to wrap the QueryEngine as a tool and let an agent call it. For more on components, the LlamaIndex documentation has Components Guides and a Guide on RAG^[1].

Adapted from lesson content on LlamaIndex components in unit 2 of Hugging Face's open-source course *Agents Course*^[1], released under the Apache-2.0 License.