TellWorldAI World Archive
Ask中文

Home›Practice›Practice

Practice

Getting started with EmbeddingGemma 2 in Sentence Transformers

EmbeddingGemma 2 maps text, images, video and audio into one 768-dimensional space. This guide covers installing Sentence Transformers, task prefixes, vector truncation and mixed-modality input.

What EmbeddingGemma 2 is and what it is for

EmbeddingGemma 2 is an open multimodal embedding model from Google DeepMind. It maps text, including code, along with images, video and audio, alone or combined, into one 768-dimensional vector space.[1] The full model has 740M parameters. It pairs a 270M text model with a 170M vision encoder and a 300M audio encoder.[1] The text part has a 130M transformer backbone and a 140M embedder.[1]

It is built to run on consumer hardware such as phones and laptops, which suits on-device search, retrieval-augmented generation (RAG), classification and clustering.[1] It understands more than 100 languages and improves on its predecessor by about 14% on code tasks.[1] The context window is 8,192 tokens, enough for minutes of audio or video.[1]

Install and run a first search

Install Sentence Transformers and Transformers:[1]

pip install -U sentence-transformers transformers

The following example encodes a query and a document, then scores them:[1]

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("google/embeddinggemma-2")

query = "What causes the northern lights?"
document = "The northern lights are caused by charged particles from the sun."

query_emb = model.encode(query, prompt_name="SearchQuery")
doc_emb = model.encode(document, prompt_name="Document")
print(model.similarity(query_emb, doc_emb))

The query uses the SearchQuery prompt and the document uses the Document prompt. model.similarity returns the score between the two vectors.[1]

Why task prefixes matter

The model was trained with short task instruction prefixes placed in front of text. A correct prefix improves quality. Without one, the model still works, but precision drops.[1] Prefixes apply to text only. Images, video and audio go in without a prefix.[1]

Search is an asymmetric task, so queries and documents get different prefixes. A query takes task: search result | query: {query}. A document takes title: {title} | text: {content}, and title: none when there is no title.[1] Classification, clustering and sentence similarity are symmetric tasks, so every input in the comparison gets the same prefix.[1]

The library ships prompt names for each case: SearchQuery, QuestionAnswering, FactChecking, CodeRetrieval, Classification, Clustering and SentenceSimilarity.[1] Watch out for prompt_name="Document", which applies title: none. A document with a real title has to be formatted by hand, for example model.encode(f"title: {title} | text: {document}").[1]

Load only the encoders you need

The vision and audio encoders are independent. If you only need text, or text with one other modality, you can turn off the unused encoders with config_kwargs and save memory.[1] Text only uses {"vision_config": None, "audio_config": None}, which is about 270M effective parameters. Text with images uses {"audio_config": None} for about 440M. Text with audio uses {"vision_config": None} for about 570M. Full multimodal uses {} for the full 740M.[1] How to omit encoders varies between libraries, so check the documentation of the library you use.[1]

Shorten vectors with Matryoshka truncation

The model is trained with Matryoshka Representation Learning, so you can keep only the first 128, 256 or 512 dimensions of the 768-dimensional output.[1] On MTEB multilingual v2, the mean task score is 61.36 at 768 dimensions, 61.17 at 512, 60.41 at 256 and 57.89 at 128.[1] Quality stays close to lossless down to 256 dimensions. At 128 dimensions, multimodal quality drops a lot, so test on your own data first.[1]

After truncation, the vector is no longer unit length. Re-normalize it before you compare vectors with cosine similarity. Skipping this step does not raise an error. The rankings just get worse.[1] Queries and documents must also share the same dimension. A 768-dimensional query cannot be scored against a 128-dimensional corpus.[1]

You can avoid the manual steps by passing truncate_dim and normalize_embeddings to model.encode():[1]

query_emb = model.encode(
   query,
   truncate_dim=128, # or 512, 256
   normalize_embeddings=True,
)

Use bfloat16 or float32, not float16

The model's activations exceed what float16 can represent. In float16, the model returns NaN or quietly degraded embeddings, and it does not raise an error, so the problem is easy to miss.[1] bfloat16 has the same 8-bit exponent as float32 and uses half the memory of float32. It is the recommended choice on hardware that supports it. Most CPUs should use float32.[1]

The following snippet picks the dtype at run time, assuming torch and SentenceTransformer are already imported:[1]

dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float32

model = SentenceTransformer("google/embeddinggemma-2", model_kwargs={"torch_dtype": dtype})

Mix text, images and video in one input

A single input can combine text with images, video and audio. Everything shares one 8,192-token context.[1] Placeholder tokens in the text mark where each media item goes: <|image|> for images, <|video|> for video and <|audio|> for audio. Each placeholder is filled, in order, from the matching key.[1] The example below encodes a product listing for search:[1]

emb_interleaved = model.encode({
    "text": "Waterproof running shoes. <|image|> Featuring a breathable mesh upper. <|image|> Grip test on wet rock: <|video|>",
    "image": ["shoe.jpg", "mesh.jpg"],
    "video": "demo.mp4",
})

The call returns one embedding for the whole input. It can be compared with any other EmbeddingGemma 2 embedding, including a text-only query such as waterproof shoes for trail running.[1]

Each modality uses a fixed share of the context. Text costs one token per subword. An image costs 280 tokens by default, which is about 29 images. A video frame costs 140 tokens by default, which is about 58 frames. Audio costs 25 tokens per second, which is about 327 seconds.[1] Mixing modalities leaves less room for each one.[1]

The vision token budget can be set anywhere from 70 to 1,120 soft tokens. A smaller budget lets you fit up to about 114 images or frames.[1] By default, video is sampled at 1 frame per second, and audio should be mono at 16 kHz.[1]

Limits and safety

EmbeddingGemma 2 is a pre-trained embedding model. It has not gone through post-training alignment, safety tuning or output moderation.[1] Because it outputs vectors instead of text, safety depends on how you use those vectors. Add application-level safeguards such as retrieval filtering and fairness testing.[1] Deployments must follow the Gemma Prohibited Use Policy.[1]

The model covers more than 100 languages, but performance may vary between them.[1] Leaving out the recommended task prefix for text tasks also lowers quality.[1]

Next steps

The technical report, arXiv 2610.37364, has more detail on the model's design.[1]

Adapted from Google's *EmbeddingGemma 2* model card on Hugging Face[1], released under the Apache-2.0 License.

Sources

  1. 1
    google/embeddinggemma-2Google (Hugging Face model card)

Cite this page: Getting started with EmbeddingGemma 2 in Sentence Transformers. TellWorld, https://www.tellworld.net/practice/practice-embeddinggemma-2-multimodal. Updated 2026-11-18

Related news

View all