Positioning
HELM is the eval framework that came out of the Stanford CRFM paper of the same name. The thesis is that one number per benchmark is not enough; a useful evaluation measures multiple metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) across a wide set of scenarios, and reports the full matrix 1. The HELM site lists 22 public leaderboards (as of October 2026), from Capabilities and Safety to AIR-Bench, MedHELM, Long Context, VHELM (vision-language), HEIM (image generation), Audio, and language-specific suites such as SEA-HELM and Arabic 2.
HELM entered maintenance mode on June 1, 2026. The code and leaderboards stay available, but the maintainers will add no new features and no new evaluations to the leaderboards. Pull requests need an attached issue and are limited to notable models, notable scenarios, fixes for external API breaks, and significant bugs 3. v0.5.16 (April 30, 2026) is still the latest release as of October 2026 4.
The framework is heavier than lm-evaluation-harness and aimed at producing reports rather than single-number scores. For practitioners, it is most useful when you need a structured comparison across many scenarios and dimensions, or the published per-instance outputs, not when you want a quick MMLU number.
Strengths
- Structured scenario-and-metric matrix. The HELM design forces multi-dimensional reporting (accuracy plus calibration plus robustness plus fairness) rather than collapsing everything to one accuracy column.
- A large public record. The leaderboards publish per-scenario, per-metric results with the underlying prompts and responses. Do not expect them to track new models: the latest Capabilities (v1.15.0) and Safety (v1.17.0) releases are both dated November 24, 2025, the newest models in Capabilities are GPT-5.1 and Gemini 3 Pro Preview from that month 2, and the maintenance-mode policy rules out new leaderboard evaluations 3. Use them as a historical baseline and a sanity check on your own harness, not to rank 2026 models.
- Apache 2.0 and reproducible. The implementation is permissively licensed, the scenario definitions are versioned, and the reported numbers are reproducible from the same harness.
Limitations
- Heavier than lm-evaluation-harness. Setup, scenario configuration, and metric stacking are more involved. For a quick MMLU comparison, the harness is simpler 5.
- Academic posture. The framework is designed for research reports, not CI. Most practitioners use it for one-off benchmarking, not as a continuous regression harness.
- No application-level coverage. HELM scores models on closed-form scenarios. RAG, agents, and prompt-level evals are out of scope; use a different tool for those.
- Maintenance mode. Upstream turns away new features and non-notable models and scenarios, and there is no fixed release cadence 3. If the scenario you need is niche, budget to maintain a fork yourself.
Best fit
Reproducing or extending published HELM numbers, borrowing its scenario definitions and multi-metric reporting design, and language-coverage and fairness audits on models HELM already supports. Strongest fit for research engineers who need HELM-comparable results. Do not start a new continuous benchmarking program on it.
Getting started
pip install crfm-helm
# Run a small subset of MMLU scenarios on a HuggingFace model.
helm-run \
--run-entries mmlu:subject=college_chemistry,model=huggingface/llama-3.1-8b-instruct \
--suite my_audit_v1 \
--max-eval-instances 100
# Summarize results into a report.
helm-summarize --suite my_audit_v1
# Optionally serve a local leaderboard UI.
helm-server --suite my_audit_v1
The local UI mirrors the layout of the public HELM leaderboards, with per-scenario and per-metric breakdowns.
Pricing notes
HELM is free under the Apache 2.0 license. Compute costs (model inference) are paid to the underlying providers. Running the full HELM scenario matrix against a frontier model is non-trivial on inference cost; most practitioners pick a subset of scenarios and a --max-eval-instances cap that produces statistically meaningful numbers without buying every token.
Alternatives
The maintenance-mode policy itself points to five open-source alternatives: Evalchemy, Inspect Evals, Lighteval, lm-evaluation-harness, and Unitxt 3. Consider lm-evaluation-harness for a lighter harness when you want a single benchmark number rather than the full matrix. Consider Inspect AI for agentic and sandboxed evals beyond closed-form scenarios. For application-level evals (RAG, agents), HELM is not the right tool; pick DeepEval, Ragas, or Promptfoo.
Citations and last verified
Verified 2026-10-02 against the HELM repository, README, and releases, the HELM quick start and maintenance-mode policy, PyPI, and the CRFM leaderboard metadata and release summaries.