Most public RAG literature evaluates English-only systems. Most production RAG systems do not serve English-only users. The gap is where teams discover, post-launch, that the embedding model collapses synonyms in Hindi, that the faithfulness judge slips into English when reasoning about Vietnamese, and that the answer reads as a translation rather than as native prose. This page lays out the dimensions you have to evaluate separately when the language is not English.
What changes outside English
Three things change. Embedding-model behavior varies across languages, sometimes drastically. Tokenization shifts: a token that costs one in English costs three or four in Devanagari or Chinese, changing both cost and effective context. Judge behavior shifts: a judge calibrated against English faithfulness labels does not generalize to other languages, and may default to scoring the English back-translation of the output rather than the output itself. Each shift is a separate failure surface, and a single end-to-end pass rate hides all three.
Figure: The three failure surfaces of non-English RAG evaluation: per-language embedding retrieval (a model averaging 60 on English can land at 40 on the target language), LLM judge calibration (92 percent human agreement on French vs 78 percent on Indonesian), and tokenization cost (the same sentence costs 3-4x more tokens in Hindi or Chinese). Each surface needs its own per-language number.
Retrieval: embedding and lexical
MTEB tracks per-language retrieval scores for most production embedding models, and the per-language deltas are large. A model averaging 60 on English may land at 40 on the target language; a multilingual-from-scratch model may invert the ranking [1]. MMTEB, the multilingual extension, covers more than 500 tasks across more than 250 languages, and the leaderboard carries it as MTEB(Multilingual, v2) [2][1]. Use it to build the shortlist, then score your candidate embedding models on a labeled retrieval set in the target language, not on the English benchmark. The labeled set can be small (a hundred queries with gold chunks) and still distinguish a model that handles the language from one that does not. Synthetic queries are a weak substitute here: the Evals FAQ lists low-resource languages and dialects as a case where LLM-generated samples are often unrealistic [3], so have native speakers label real queries (see synthetic query generation).
# embed_compare.py
from sentence_transformers import SentenceTransformer
import numpy as np
CANDIDATES = [
"intfloat/multilingual-e5-large",
"BAAI/bge-m3",
"Alibaba-NLP/gte-multilingual-base",
]
def recall_at_k(model_name, queries, chunks, k=5):
model = SentenceTransformer(model_name)
chunk_embs = model.encode([c["text"] for c in chunks], normalize_embeddings=True)
ids = [c["id"] for c in chunks]
hits = 0
for q in queries:
q_emb = model.encode([q["query"]], normalize_embeddings=True)[0]
gold = ids.index(q["gold_chunk_id"])
top = np.argsort(-(chunk_embs @ q_emb))[:k]
hits += int(gold in top)
return hits / len(queries)
for name in CANDIDATES:
print(f"{name}: Recall@5 = {recall_at_k(name, queries, chunks):.3f}")
Same queries, same chunks, three multilingual models, one number per model. The winner on the target language is the one you ship, regardless of where it ranks on English-only leaderboards. Two open candidates worth adding to the list: Qwen3-Embedding (0.6B, 4B, and 8B sizes, 100-plus languages) and EmbeddingGemma (300M parameters, 100-plus languages) [4][5]. Check each candidate's loading instructions before you run the loop: the gte-multilingual-base model card loads it with trust_remote_code=True, which executes code shipped with the model, and the loop above does not pass that flag. Read the shipped code and then pass the flag for that model, or leave the model out [6].
Lexical retrieval (BM25, hybrid) deserves a second look: BEIR found BM25 a robust zero-shot baseline [7]. Languages with rich morphology (Hungarian, Turkish, Russian) interact badly with naive tokenization, so configure a language-aware analyzer (stemming, lemmatization, compound splitting) and measure its effect on Recall@K against the same labeled set.
Generation: faithfulness and answer relevance
Faithfulness in non-English is the same atomic-fact check, run in the target language. The pitfall is using an LLM judge that translates the output to English before scoring. The translation step adds noise, and the resulting score reflects translation quality as much as faithfulness. Pin the judge prompt to the target language and check, on a labeled subset, that the judge agrees with a native-speaker reviewer above 90 percent [8]. Calibration is per-language: a judge hitting 92 percent on French may sit at 78 percent on Indonesian. Published work agrees. Fu and Liu ran five judge models on five tasks in 25 languages and measured an average Fleiss' kappa of about 0.3 across languages, with the weakest results in low-resource languages and no reliable gain from larger models or multilingual training; they propose ensembling several judges to steady the verdicts [9]. Put a reviewer who understands both the language and the culture of your users in charge of the error analysis, because understanding the literal meaning of an answer is not enough to judge whether its tone fits [3].
Answer relevance has a related failure mode: the model produces an answer in a different language than the question. Track the per-query language match. A few-percent rate of off-language responses is often invisible to an English-speaking dev team and a major complaint in the target market.
Two public sets give you non-English RAG queries to borrow as a floor. MIRAGE-Bench covers 18 languages, scores answers in an arena format, and uses a surrogate judge that tracks GPT-4o judgments at a Kendall tau of 0.909 [10]. NoMIRACL covers 18 languages and measures what happens when no retrieved passage is relevant: in the paper's runs Llama-2 and Orca-2 hallucinated on more than 88 percent of the non-relevant subset, while Mistral and Llama-3 failed to recognize the relevant passage up to 74.9 percent of the time [11]. Add your own negative queries in the target language; the hallucination and miss rates trade against each other.
Tokenization and cost
A Hindi sentence may cost three or four times as many tokens as the equivalent English sentence on most tokenizers. A chunk size of 500 tokens that holds 150 useful English concepts holds 70 in Hindi. Retune chunking per language, often by counting in characters or approximate concepts rather than tokens. Report per-language costs in the scorecard: a retrieval architecture that looks affordable on English benchmarks may be three times the cost in the target language. Count tokens with the tokenizer of the exact model you ship, not a rule of thumb. A NeurIPS 2023 study found that the same text can differ by up to 15 times in token count across languages [12], and tokenizers change between model generations: as of October 2026, Anthropic's pricing page says Claude 4.7 and later models produce about 30 percent more tokens for the same text, with the exact increase depending on the content [13].
Translation as a layer
For some products the right answer is to translate user queries to English, retrieve against an English corpus, and translate the answer back. The WMT shared task tracks machine translation quality per language pair. WMT24 titled its findings with the claim that the LLM era is here but MT is not solved yet [14]. WMT25 covered 30 language pairs with difficulty-sampled test sets built from more complex source documents, and its findings title says it is time to stop evaluating on easy test sets [15]. Test your language pair on your own domain before you treat translation as a solved layer. The trap is silent: a 5 percent translation error rate per direction stacks to a 10 percent end-to-end floor before any retrieval or generation error. Evaluate translation as its own layer.
CAUTION
Translating the eval set to English to reuse English judges produces misleading numbers. The eval has to run in the user's language, with judges calibrated in that language, or scores will not predict user-visible quality.
The minimum non-English scorecard has six numbers per language: Recall@K, faithfulness, answer relevance, language-match rate, per-query cost, and judge-against-human agreement. All six get reported per release alongside the English numbers. A regression in any one is a release blocker [3].
When the system moves from answering questions to taking actions, the scorecard has to cover whole trajectories rather than single responses; that is the territory of Agentic Evals.