Positioning
The lm-evaluation-harness, often shortened to lm_eval, is EleutherAI's open-source benchmark runner. It was the backend for the Hugging Face Open LLM Leaderboard and has been used in hundreds of papers 1. The leaderboard Space is now labeled Archived, so treat its scores as a frozen record of older open models, not a live ranking 2. The harness covers MMLU, BBH, ARC, HellaSwag, TruthfulQA, GSM8K, HumanEval, GPQA, IFEval, MMMLU, and hundreds of other tasks via YAML configs. The December 2025 refactor split the CLI into run, ls, and validate subcommands and added YAML run configs via --config; the old single-command form still works. The base pip install no longer pulls model backends, so install the extra you need (hf, vllm, api) 1.
The harness is the right tool when you want to compute a model's score on a published benchmark with the same prompt template, few-shot examples, and scoring rules used by the academic community. It is one implementation among several, though. Hugging Face measured LLaMA 65B on MMLU at 0.636 with the original implementation, 0.637 with HELM's, and 0.488 with the harness as of January 2023, so a number only compares to another produced by the same implementation and version 3 4.
Strengths
- De facto academic standard. Hundreds of papers report numbers from this harness, so reproducing a published number usually comes down to matching the harness version, task config, and few-shot count.
- YAML-first tasks. Adding a new benchmark or variant is a YAML file, not a code change, and
lm-eval validate checks the config before you spend GPU hours. In September 2026 the main branch added plugins: model backends, filters, metrics, and aggregations register from your own package through an lm_eval.* entry point instead of a fork. It landed after v0.4.13, so it needs a source install until the next release 1.
- Wide model backend support. HuggingFace Transformers, vLLM, OpenAI-compatible APIs, Anthropic, local quantized models, and others all plug in via simple model adapters.
Limitations
- Not application-level. The harness scores models on closed-form benchmarks. It does not help you eval a RAG pipeline, an agent, or a prompt under your custom rubric.
- Format sensitivity. Small changes to prompt templates, normalization, and choice-letter formatting can move benchmark scores by several points. Independent analyses have flagged this as a real reproducibility issue 4. Harness fixes move them too: v0.4.13 (August 31, 2026) fixed eval documents leaking into their own few-shot prompts, a multiple-choice regex filter that scored some correct answers as wrong on BBH, and group standard errors up to about 3x too narrow for groups set to
weight_by_size: false 5. Record the harness version next to every score and re-run baselines after an upgrade.
- Chat APIs only run generative tasks. Hosted chat endpoints return no prompt log-probabilities, so
openai-chat-completions and the Anthropic backends support generate_until tasks only. Multiple-choice tasks such as MMLU, ARC, and HellaSwag need a local model or a completions endpoint that returns logprobs, such as a vLLM server 6.
- No observability or dashboards. Results are JSON files. Pair with a hub or push to a notebook for visualization.
Best fit
Pure model benchmarking when you swap base models, contamination-style ablations, language-coverage audits, and reproducing published numbers. Strongest fit for research engineers and platform teams that need to make data-driven model selection decisions.
Getting started
pip install "lm_eval[hf,api]"
# MMLU on a local HuggingFace model.
lm_eval \
--model hf \
--model_args pretrained=meta-llama/Llama-3.1-8B-Instruct \
--tasks mmlu \
--batch_size auto \
--output_path results/llama_mmlu.json
# Or on a vLLM server's OpenAI-compatible completions endpoint.
lm_eval \
--model local-completions \
--model_args model=meta-llama/Llama-3.1-8B-Instruct,base_url=http://localhost:8000/v1/completions \
--tasks arc_challenge,hellaswag \
--num_fewshot 5
Earlier versions of this example pointed openai-completions at gpt-4o-mini. That backend calls OpenAI's legacy Completions endpoint, which does not serve gpt-4o-mini 7, and ARC and HellaSwag need log-probabilities that chat endpoints do not return. The harness README recommends serving large models through vLLM's OpenAI-compatible API and evaluating them with local-completions, which keeps log-likelihood tasks available 1.
The JSON output reports per-task accuracy, per-category breakdowns, and standard-error estimates. The CLI has no run-diff command, so compare runs in a notebook.
Pricing notes
lm-evaluation-harness is free under the MIT license. Compute costs (HuggingFace inference, vLLM hosting, OpenAI or Anthropic API calls) are paid to the underlying providers. API spend only buys generative tasks, since chat APIs cannot run the log-likelihood ones; local vLLM on a multi-GPU node is the usual cost-controlled path for repeated runs.
Alternatives
Consider HELM for a more structured scenario-and-metric matrix, but it entered maintenance mode on June 1, 2026 and its leaderboards get no new evaluations 8. Consider Inspect AI for agentic and sandboxed evals that go beyond closed-form benchmarks. OpenAI Evals is a weak pick for new work: the open-source repo is in maintenance only and the hosted platform shuts down on November 30, 2026 9. For application-level evals (RAG, agents), this harness is the wrong tool; pick DeepEval, Ragas, or Promptfoo.
Citations and last verified
Verified 2026-10-02 against the lm-evaluation-harness README, CLI reference, API guide, and v0.4.13 release notes, the Open LLM Leaderboard Space, the Hugging Face MMLU post, and the OpenAI gpt-4o-mini model page.