Building a search app with embeddings

Search is the quiet workhorse of any content library, and embeddings upgrade it from keyword matching to meaning matching: a user types a question, and the application returns the exact spot in a video where the answer is spoken. This page builds such a semantic search over a YouTube lecture library, from the embedding pipeline to the Azure resources it needs.

Semantic search versus keyword search

Semantic search uses the meaning of the words in a query to return relevant results ^[1]. Searching "my dream car" is understood as looking for an ideal car, not literally dreaming about one; keyword search would match text about dreams and cars and often return irrelevant results ^[1]. That difference is the whole reason to reach for embeddings: users type questions, not keywords, and semantic search answers the question they meant.

The concrete goal here: a student types "What are Jupyter Notebooks?" or "What is Azure ML" and gets back relevant YouTube videos — including a URL with a timestamp that jumps straight to the place in the video where the answer is located ^[1].

What a text embedding is

Text embeddings are semantic numerical representations of text, a technique from natural language processing; a machine compares these numbers far more easily than raw words ^[1]. Passing "Today we are going to learn about Azure Machine Learning." to the OpenAI Embedding API returns a vector of 1536 numbers, each representing a different aspect of the text ^[1]:

[-0.006655829958617687, 0.0026128944009542465, 0.008792596869170666, -0.02446001023054123, -0.008540431968867779, 0.022071078419685364, -0.010703742504119873, 0.003311325330287218, -0.011632772162556648, -0.02187200076878071, ...]

Building the index: a five-step pipeline

The embedding index for the video library is built by a script pipeline ^[1]:

1. Download the transcript for each video in the playlist.
2. Use OpenAI Functions to extract the speaker name from the first 3 minutes of each transcript.
3. Chunk the transcripts into 3-minute text segments, with about 20 words overlapping from the next segment so an embedding is not cut off mid-thought and search context is better.
4. Summarize each segment into 60 words with the OpenAI Chat API.
5. Pass each segment to the OpenAI Embedding API and store the segment with its 1536-number vector in an index file named embedding_index_3m.json.

For a small library, the index can simply be a JSON file loaded into a Pandas DataFrame; in production it belongs in a vector database such as Azure Cognitive Search, Redis, Pinecone, or Weaviate ^[1]. The overlap detail in step 3 deserves attention — a segment boundary in the middle of an explanation produces embeddings that match nothing well, and the 20-word overlap is what prevents that ^[1].

Finding the answer: cosine similarity

With every segment embedded, search becomes a nearest-neighbor problem. Cosine similarity measures how similar two vectors are: vectorize the user's question with the Embedding API, compute the cosine similarity between the query vector and every vector in the index, sort the results, and the highest-scoring segments are the most similar to the question ^[1].

Why cosine rather than plain distance? It measures the cosine of the angle between two vectors in a multidimensional space, so two documents that are far apart by Euclidean distance because of their size can still have a small angle — and high similarity — if they discuss the same thing ^[1].

Azure resources for the embedding model

The search app itself runs on Python 3.10 or later on Windows 11, macOS, or Ubuntu 22.04 ^[1]. The embedding model runs as an Azure OpenAI deployment, provisioned from the Azure portal's Cloud Shell. First the resource group ^[1]:

az group create --name semantic-video-search --location eastus

Then the Azure OpenAI Service resource ^[1]:

az cognitiveservices account create --name semantic-video-openai --resource-group semantic-video-search \
    --location eastus --kind OpenAI --sku s0

Grab the endpoint and key the application will use ^[1]:

az cognitiveservices account show --name semantic-video-openai \
   --resource-group  semantic-video-search | jq -r .properties.endpoint
az cognitiveservices account keys list --name semantic-video-openai \
   --resource-group semantic-video-search | jq -r .key1

And deploy the embedding model itself ^[1]:

az cognitiveservices account deployment create \
    --name semantic-video-openai \
    --resource-group  semantic-video-search \
    --deployment-name text-embedding-ada-002 \
    --model-name text-embedding-ada-002 \
    --model-version "2"  \
    --model-format OpenAI \
    --sku-capacity 100 --sku-name "Standard"

From there, the application loop is short: take the user's question, embed it with the deployed model, compute cosine similarity against the index, and return the top videos with their timestamps. When the solution notebook runs, the user is prompted for a query in an input box ^[1].

Adapted from lesson 8 of Microsoft's open-source course *Generative AI for Beginners*^[1], released under the MIT License.