RAG and vector databases

A chatbot only knows two things: its training data and whatever you put in the prompt. Anything newer, private, or domain-specific is out of reach — and asking the model to guess produces fabrications. Retrieval Augmented Generation (RAG) fixes this by feeding the model relevant passages retrieved from your own knowledge base. This page builds one end to end: chunking, embeddings, vector search, and the final grounded chatbot.

Why RAG

GPT-4's knowledge cutoff is September 2021, and training data excludes confidential material such as personal notes or product manuals ^[1]. RAG addresses both gaps in four steps ^[1]:

1. Knowledge base: documents are ingested and preprocessed — split into smaller chunks, transformed into text embeddings, stored in a database.
2. User query: the user asks a question.
3. Retrieval: the embedding model fetches relevant information from the knowledge base to incorporate into the prompt.
4. Augmented generation: the LLM enhances its response based on the retrieved data.

Architecturally, RAG combines a retriever with a generator: the query becomes a vector, is matched against the document index for relevant chunks, and those chunks go to the LLM as context. The retriever and generator can be different models, and RAG does not require an encoder-decoder transformer ^[1]. The research paper that proposed RAG describes two approaches: RAG-Sequence, using retrieved documents to predict the best answer to a query, and RAG-Token, generating token by token with retrieved documents ^[1].

Three reasons to prefer RAG over alternatives ^[1]: information richness, since responses stay current and domain-specific through the internal knowledge base; reduced fabrication, because the context comes from verifiable data; and cost, since it is more economical than fine-tuning an LLM.

Vector databases and chunking

A vector database stores, manages, and searches embedded vectors — numerical representations of documents — which is what makes semantic processing feasible for AI systems ^[1]. Because LLMs cap input tokens, documents are split into chunks before embedding; the chunks most similar to the query get returned with the prompt, and chunking also cuts the token count and cost of every call ^[1]. Popular choices include Azure Cosmos DB, Clarifyai, Pinecone, Chromadb, ScaNN, Qdrant, and DeepLake ^[1]. An Azure Cosmos DB account comes together with four CLI commands ^[1]:

az login
az group create -n <resource-group-name> -l <location>
az cosmosdb create -n <cosmos-db-name> -r <resource-group-name>
az cosmosdb list-keys -n <cosmos-db-name> -g <resource-group-name>

Chunking granularity matters more than it looks. Chunks can be sentence- or paragraph-level, and since a chunk's meaning comes from surrounding words, adding context helps — the document title, or text before and after the chunk ^[1]. A minimal word-count chunker shows the mechanics ^[1]:

def split_text(text, max_length, min_length):
    words = text.split()
    chunks = []
    current_chunk = []

    for word in words:
        current_chunk.append(word)
        if len(' '.join(current_chunk)) < max_length and len(' '.join(current_chunk)) > min_length:
            chunks.append(' '.join(current_chunk))
            current_chunk = []

    # If the last chunk didn't reach the minimum length, add it anyway
    if current_chunk:
        chunks.append(' '.join(current_chunk))

    return chunks

For embeddings, models include word2vec, OpenAI's ada-002, and Azure Computer Vision; pick by language support, content type (text, images, audio), input size, and embedding output length ^[1].

Retrieval and vector search

When a user asks a question, the retriever encodes it into a vector, searches the index for related document vectors, and converts both back into text for the LLM ^[1]. Three search modes exist: keyword search for text, vector search for semantic search by meaning, and hybrid, which combines both ^[1]. One pitfall: when nothing similar exists in the database, the system still returns the best information it can — guard against this with a maximum-distance threshold or hybrid search ^[1].

Similarity is usually measured with cosine similarity, based on the angle between two vectors; Euclidean distance and dot product are alternatives ^[1]. A search index makes lookup fast even in a large database; a local one is a few lines with scikit-learn ^[1]:

from sklearn.neighbors import NearestNeighbors

embeddings = flattened_df['embeddings'].to_list()

# Create the search index
nbrs = NearestNeighbors(n_neighbors=5, algorithm='ball_tree').fit(embeddings)

# To query the index, you can use the kneighbors method
distances, indices = nbrs.kneighbors(embeddings)

Results can then be reordered by relevance — a reranking LLM sorts them from most relevant, and Azure AI Search does this automatically with a semantic reranker ^[1]:

# Find the most similar documents
distances, indices = nbrs.kneighbors([query_vector])

# Print the three most similar documents with their distance to the question
for distance, index in zip(distances[0][:3], indices[0][:3]):
    print(flattened_df['chunks'].iloc[index])
    print(flattened_df['path'].iloc[index])
    print(distance)

Putting it together

The final chatbot embeds the question, retrieves the nearest chunks, stacks them as context, and generates the answer ^[1]:

user_input = "what is a perceptron?"

def chatbot(user_input):
    # Convert the question to a query vector
    query_vector = create_embeddings(user_input)

    # Find the most similar documents
    distances, indices = nbrs.kneighbors([query_vector])

    # add documents to query  to provide context
    history = []
    for index in indices[0]:
        history.append(flattened_df['chunks'].iloc[index])

    # combine the history and the user input
    history.append(user_input)

    # create a message object
    messages=[
        {"role": "system", "content": "You are an AI assistant that helps with AI questions."},
        {"role": "user", "content": "\n\n".join(history) }
    ]

    # use the Responses API to generate a response
    response = client.responses.create(
        model="gpt-5-mini",
        max_output_tokens=800,
        input=messages,
        store=False,
    )

    return response.output_text

chatbot(user_input)

Evaluate the result along four axes: response quality (natural and fluent), groundedness (did the answer come from the supplied documents), relevance (does it match the question), and fluency (is it grammatical) ^[1]. Typical applications include employee Q&A over company data, recommendation systems, chatbots that personalize from chat history, and image search for recognition and anomaly detection ^[1]. Frameworks like Semantic Kernel, LangChain, and AutoGen take care of most of this plumbing for you ^[1].

Adapted from lesson 15 of Microsoft's open-source course *Generative AI for Beginners*^[1], released under the MIT License.