Positioning
OpenAI Evals is the original open-source evaluation framework from OpenAI, released in 2023. The codebase is a registry of YAML- and JSON-defined evals plus Python templates for common patterns (basic match, model-graded, multiple-choice), run locally through the oaieval CLI 1. The repo is public and not archived, but it is in maintenance only: the last change to eval code (November 2025) removed a suite whose dependencies had died, the only 2026 commits pin CI dependencies, and the evals package on PyPI was last released in May 2024.
The hosted Evals platform in the OpenAI dashboard is a different product. It defines an eval through its own API objects (data_source_config, testing_criteria, graders), not the repo's YAML registry 2. OpenAI announced its deprecation on June 3, 2026: existing evals become read-only on October 31, 2026, and the dashboard and API are scheduled to shut down on November 30, 2026 3. OpenAI's migration guide recommends Promptfoo and maps graders to Promptfoo assertions; it describes no export or conversion tool, so plan to recreate each eval by hand in promptfooconfig.yaml 4. The repo README still points readers to the hosted dashboard, and the Cookbook walkthrough recommends the hosted product over the repo, so read both with the shutdown in mind 5.
Treat the repo as a library of reference eval implementations, not a CI dependency. For application-level evals (RAG, agents, custom rubrics), DeepEval, Promptfoo, and Inspect AI are more ergonomic and actively maintained.
Strengths
- Large reference registry. The repository has 19.5k stars and 3.1k forks (as of October 2026) and a long tail of community-contributed evals across reasoning, factuality, code, and persona benchmarks. Reading how a model-graded eval is specified there is a fast way to design your own.
- Template-first authoring. A basic or model-graded eval needs only JSONL data and a YAML entry, no Python 1.
- Runs without the hosted platform.
oaieval calls the model API directly and writes results to a local file, so the November 2026 shutdown does not touch it.
Limitations
- Maintenance only. No feature work since 2024, and the README says the maintainers are not accepting evals with custom code. Expect to patch it yourself.
- Model routing predates GPT-4o.
oaieval sends a bare model name to Chat Completions only when it is gpt-3.5-turbo*, gpt-4, or gpt-4-*; any other OpenAI model ID falls through to the legacy Completions endpoint. Current chat models such as gpt-4o-mini are not served there 6, and the gpt-3.5-turbo, gpt-4, and gpt-4-turbo snapshots the router does recognize shut down on October 23, 2026 3. Register newer models as completion functions, as in the example below 7.
- Not where OpenAI's published numbers come from. The accuracy numbers OpenAI published alongside its models came from a separate repo, simple-evals, which stopped updating for new models in July 2025 and now hosts only the HealthBench, BrowseComp, and SimpleQA reference implementations 8. Neither repo tracks current OpenAI model releases.
- OpenAI-centric. Non-OpenAI models can be wired in through completion functions, but the templates and docs assume OpenAI is the primary target.
- Vendor self-interest. OpenAI runs and maintains this framework; vendor-published benchmarks on OpenAI models are vendor-published.
Best fit
Reading and borrowing reference eval designs, keeping an existing oaieval suite alive while you migrate, and small offline checks on OpenAI models. If you built evals on the hosted platform, record their definitions before October 31, 2026 and rebuild them in Promptfoo or another harness before the November 30 shutdown.
Getting started
git clone https://github.com/openai/evals.git
cd evals
pip install -e .
# Register gpt-4o-mini as a Chat Completions function;
# the bare name would be routed to the legacy Completions API.
mkdir -p my_registry/completion_fns
cat > my_registry/completion_fns/chat.yaml <<'YAML'
chat/gpt-4o-mini:
class: evals.completion_fns.openai:OpenAIChatCompletionFn
args:
model: gpt-4o-mini
YAML
# Run a registered eval against it.
oaieval chat/gpt-4o-mini test-match \
--registry_path my_registry \
--record_path results/test_match.jsonl
The --registry_path mechanism is the documented way to add completion functions without editing the repo 7. The OpenAI Cookbook walkthrough covers dataset construction, eval templates, and reading results, but it opens by recommending the hosted product that is shutting down 5. For a custom eval, write a YAML registry entry that points at a JSONL dataset and one of the templates; add a Python class only when no template fits.
Pricing notes
The framework is free under the MIT license. Compute is billed at OpenAI API rates per token, and hosted dashboard runs bill the same way until the platform shuts down on November 30, 2026 3. Model-graded evals (the bulk of the more interesting templates) call a judge model per row, which is the dominant cost; cache aggressively and use a cheaper judge model when calibration shows it agrees with the gold judge within the tolerance you care about. oaieval caches requests by default (--cache), and --max_samples caps a run while you iterate.
Alternatives
Promptfoo is OpenAI's own recommended destination for hosted-platform users, and the obvious pick for application-level CI regression 4. Consider DeepEval for pytest-style suites. Consider Inspect AI for sandboxed agentic evals. Consider lm-evaluation-harness for academic benchmarks across many model providers. HELM has a structured scenario-and-metric matrix, but it has been in maintenance mode since June 2026.
Citations and last verified
Verified 2026-10-02 against the OpenAI Evals repository (README, commit history, registry.py, completion-function docs), PyPI, the OpenAI Evals guide, the OpenAI deprecations page, the gpt-4o-mini model page, the Promptfoo migration guide, the simple-evals README, and the OpenAI Cookbook.