Positioning
Inspect AI is the open-source frontier-model eval framework from the UK AI Security Institute (AISI) and Meridian Labs. The library treats an eval as a Python program: a dataset, a solver chain (prompts, tools, generation), and a scorer, all composed by typed primitives. A VS Code extension gives the eval program a debugger-style UI for inspecting solver traces 1. Sandboxing is a first-class concern. Docker is built in; Kubernetes, Modal, Daytona, EC2, Proxmox, Vagrant, and NVIDIA OpenShell backends install as separate packages, so code-execution evals run in isolation without ad-hoc plumbing 2.
The companion Inspect Evals collection covers reasoning, agentic, safety, multimodal, and tool-use evals. The Inspect docs advertise over 200 prebuilt evals; the Inspect Evals register itself listed 178 when fetched on 2026-10-02, 129 in the inspect_evals repo and 49 external listings that point at implementations hosted elsewhere 3. The repo is maintained by Generality Labs, a London nonprofit; it was founded in 2024 with contributions from AISI, Arcadia Impact, and the Vector Institute 4.
Figure: Inspect AI eval framework architecture: a Task pairs a dataset of Samples with a solver chain (prompts, tools, generate) whose tool calls run inside a Docker, Kubernetes, Modal, or Proxmox sandbox. A scorer such as match() grades every sample, and results land in eval logs that the VS Code extension renders as a per-sample solver trace.
Strengths
- Sandboxing as a primitive. Docker, Kubernetes, Modal, and Proxmox backends mean code-execution and agentic evals do not require bespoke harnesses. This is the rare framework where running 200 SWE-bench problems against a sandboxed Python container is a few lines of code.
- A large prebuilt eval register. Reasoning (GPQA, MATH), coding (HumanEval, MBPP), agentic (SWE-bench, GAIA, AgentBench, OSWorld), safety (AgentHarm, StrongREJECT, XSTest, WMDP), and multimodal evals live in
inspect_evals, which ships a release every week or two (v0.23.0 on October 2, 2026) 4.
- High velocity for a gov-backed project. The repository shows 7,883 commits (as of October 2026), and the changelog lists 25 releases between August 4 and October 2, 2026, up to 0.3.276. That is unusual cadence for a government-maintained tool 5 6.
Limitations
- Steeper learning curve than pytest-style frameworks. Solvers, scorers, and the task abstraction add concepts that DeepEval and Promptfoo avoid. Worth it for frontier-style evals, overkill for a quick CI smoke test.
- Python-only. No first-class TypeScript or other-language SDK. Teams whose application is TypeScript run Inspect as a sidecar.
- No hosted product. Inspect is OSS-only. Dashboards, dataset versioning, and collaborative review are not in scope; pair with a hub like Langfuse or Phoenix if you need that.
- Fast cadence moves scores. Minor releases change behavior you measure. 0.3.262 (September 2, 2026) made panels of model graders require a strict majority, so ties now come back unscored and leave the denominator;
reducer="mode" restores the old rule. 0.3.264 (September 16) runs bash_session(), text_editor(), and sandboxed MCP servers as the sandbox's default user instead of root, and 0.3.272 (September 28) deprecated the web_browser() tool 6. Pin the version and re-baseline when you bump it.
Best fit
Frontier-model evaluation, safety and red-team programs that need sandboxing, agentic eval that runs SWE-bench, GAIA, or OSWorld at scale, and academic or policy-grade reports that need reproducible eval programs. Strongest fit for research engineers and safety teams.
Getting started
from inspect_ai import Task, eval as run_eval
from inspect_ai.dataset import Sample
from inspect_ai.scorer import match
from inspect_ai.solver import generate
task = Task(
dataset=[
Sample(input="What is 2 + 2?", target="4"),
Sample(input="Capital of France?", target="Paris"),
],
solver=[generate()],
scorer=match(),
)
if __name__ == "__main__":
run_eval(task, model="anthropic/claude-sonnet-4-6")
Earlier versions of this example used claude-3-5-sonnet-20241022, which Anthropic retired on October 28, 2025, with claude-sonnet-4-6 as the listed replacement 7. For the CLI, wrap the Task in a function decorated with @task, which is how inspect eval discovers it; inspect eval inspect_eval.py --model openai/gpt-4o-mini then runs the same eval, and the VS Code extension renders the solver trace per sample 1.
Pricing notes
Inspect AI is free under the MIT license. No paid SKU. Compute costs (model calls, sandboxing) are paid to the underlying providers. Modal and Kubernetes sandboxing add infrastructure cost that scales with the number of concurrent agent rollouts; budget separately for that line item when running SWE-bench-style suites at scale.
Alternatives
Consider DeepEval or Promptfoo for pytest-style or CLI-first OSS regression. Consider lm-evaluation-harness for raw model benchmarks without sandboxing. Consider HELM for its published scenario matrix, though HELM has been in maintenance mode since June 1, 2026 and its maintainers list Inspect Evals among the alternatives 8. For agentic eval specifically, Inspect is usually the right answer. OpenAI Evals is no longer a live option for new work: its hosted platform goes read-only on October 31, 2026 and shuts down on November 30, 2026 9.
Citations and last verified
Verified 2026-10-02 against the Inspect AI documentation and sandboxing page, the inspect_ai changelog and commit history, PyPI, the inspect_evals repository and register, and the Anthropic model deprecations page.