Embedding similarity scores two pieces of text by the cosine of their embedding vectors. The embedder is a sentence-level encoder (SimCSE, BGE, E5, Qwen3-Embedding, OpenAI text-embedding-3, Gemini Embedding, Cohere Embed 5). The score lives in roughly [0.4, 1.0] for real text pairs and has operational range only on the right job.
When embedding similarity works
Three eval jobs map cleanly to cosine over sentence embeddings.
Retrieval recall in RAG. Given a query and a ranked list of retrieved chunks, embed each against a labeled gold passage; recall@K is the rate at which the gold sits in the top K. The metric is the ranking metric (recall@K, MRR, NDCG), not the cosine. The cosine is the underlying score. The retrieval metrics chapter covers the catalog.
Output deduplication. When sampling many candidates from the same prompt, near-duplicates share cosine above 0.95. Cluster on that threshold; report distinct-N. Useful for synthetic-data pipelines where lack of diversity is the failure mode.
Semantic regression detection. Compute cosine between the current run's output and the previous run's output on the same input. A drop below 0.7 flags a likely behavioral change worth a human look. A triage signal, not a quality score.
In all three jobs, the cosine value is interpreted against a calibrated threshold on the same embedder, not as an absolute quality number.
Figure: Embedding similarity thresholds on the cosine axis: a drop below 0.70 flags a semantic regression, quality scores compress into the 0.85-0.95 band where 0.88 vs 0.89 is noise, and pairs above 0.95 are near-duplicates for deduplication. Cosine thresholds hold only per embedder and per distribution, so calibrate on labeled pairs instead of reusing absolute values.
When embedding similarity fails
The seductive case that does not work: scoring generation quality by cosine to a reference. Two problems stack. The reference is often not the quality ceiling: Yan walks through the evidence that CNN/DailyMail reference summaries are outperformed by model outputs, so closeness to them rewards the wrong thing 1. And embedder cosines compress the range: against a reasonable reference, almost every reasonable output lands in a narrow band near the top of the scale, so measure that band on your own pairs before reading anything into a score. The metric loses discriminative power exactly where it would be most useful (separating a good summary from a great one).
Modern embedders are trained on contrastive objectives that pull semantically related text together. They are designed not to discriminate at the fine-grained level; that is the feature that makes them good retrievers and bad quality scorers. SimCSE scores sentence embedders on the two properties Wang and Isola tied to contrastive learning, alignment (related texts close) and uniformity (everything else spread out), and finds that pre-trained embeddings align well but are highly anisotropic: they occupy a narrow cone, which is why their cosines bunch together. Contrastive training alleviates the anisotropy 2.
The same compression undermines system ranking. If two summarization systems sit at 0.88 and 0.89 mean cosine, the ranking is within the noise floor.
Picking the embedder
MTEB is the operational reference for which embedder to use 3. Its multilingual expansion, MMTEB, covers over 500 tasks across 250+ languages, and the repository is now a toolbox that also evaluates multimodal and retrieval systems 4 3. Pick the top three on the sub-task that matches the eval job (retrieval for RAG recall, STS for semantic regression, clustering for deduplication) and benchmark them on a held-out slice of real product traffic.
| Job | MTEB sub-task | Pick from |
|---|
| RAG retrieval | Retrieval, Reranking | top 3 on the relevant language slice |
| Output deduplication | STS, Clustering | top 3 on STS |
| Semantic regression | STS | top 3 on STS in the right language |
| Quality scoring | (nothing maps cleanly) | not the right job for embeddings |
The hosted APIs ship with stable endpoints and pay-per-token pricing: OpenAI text-embedding-3 with an 8,192-token input 5, Gemini Embedding, whose latest gemini-embedding-2 maps text, images, video, audio, and documents into one space with an 8,192-token limit 6, and Cohere Embed 5 (embed-v5.0-pro and embed-v5.0-fast), which takes text and images with a 128k-token window 7. Open models (BGE, E5, Qwen3-Embedding, EmbeddingGemma) self-host on a GPU. Maximum input length ranges from 512 tokens on Cohere's v3 models and 2,048 on gemini-embedding-001 up to 128k on Cohere's embed-v4.0 and Embed 5; cross-language performance varies enormously.
Computing cosine at scale
import numpy as np
from sentence_transformers import SentenceTransformer
def cosine_matrix(texts_a, texts_b, model):
a = model.encode(texts_a, normalize_embeddings=True, batch_size=64, convert_to_numpy=True)
b = model.encode(texts_b, normalize_embeddings=True, batch_size=64, convert_to_numpy=True)
return a @ b.T
def recall_at_k(sims: np.ndarray, gold_idx: np.ndarray, k: int) -> float:
top_k = np.argsort(-sims, axis=1)[:, :k]
return float((top_k == gold_idx[:, None]).any(axis=1).mean())
For production-scale retrieval, the dot product runs inside a vector index (FAISS, hnswlib, ScaNN). Eval-time mechanics are the same; production adds an ANN approximation that needs its own recall calibration against an exact-search baseline.
Calibrate the threshold
The number "0.85" has no universal meaning. It is the threshold below which one embedder reads two texts as different on one distribution. Calibrate. Compute cosine on one hundred pairs of (current output, previous output) hand-labeled as identical, similar, or different. Plot the distribution. Pick the threshold for the desired FPR at the desired TPR. Recheck quarterly because embedder versions drift. A version change can invalidate the whole index: Google states that gemini-embedding-001 and gemini-embedding-2 vectors are incompatible and that upgrading means re-embedding all existing data 6. Pin the embedder version in the eval config and treat an upgrade as a new calibration.
Three honest uses
Where cosine over embeddings earns its keep:
As the retrieval signal in a RAG eval. Cosine over embeddings is the engine of the retriever you are measuring, and recall@K against a labeled gold passage is the eval on top of it. RAGAS 8 does not score retrieval with embeddings: its context precision and context recall use an LLM by default, and their non-LLM variants use string distance or document IDs. The RAGAS metric that does use embedding cosine, SemanticSimilarity, compares a response with a reference answer, which is the quality-scoring use this page warns against 9. The retrieval metrics chapter covers the catalog.
As a fast pre-filter before an LLM judge. When a judge call costs cents and the candidate set is large, cosine triage cuts ninety percent of trivially-different candidates before the judge sees them.
As a regression sentinel on synthetic-data pipelines. When a prompt or retrieval change drifts the output distribution, the cosine of the centroid against a held-out reference shifts. The signal triggers a human look, not an automatic decision.
In all three, cosine is a triage signal feeding a more rigorous metric. It is not the metric. Hamel and Shankar draw the same line: cosine between embeddings can measure semantic closeness in retrieval and average pairwise similarity can assess output diversity, but generic similarity metrics are not useful for judging most LLM outputs 10.
What it is not
Embedding similarity is not a hallucination detector (the embedder cannot tell a confident lie from a confident truth). It is not a faithfulness score (atomic-fact decomposition is the right pattern; see generation faithfulness). It is not a quality score (LLM-as-Judge is the right pattern). It is a fast, calibrated, distribution-bound signal of semantic proximity, and used on the right job it earns its place in the pipeline 11 12.