Positioning
Harbor is a Python framework for running any agent on any task in any sandbox, in parallel 1. It is built by the team behind Terminal-Bench, and the Terminal-Bench README tells new users to install Harbor to run the benchmark 2. The code lives at harbor-framework/harbor under Apache-2.0; the older laude-institute/harbor URL redirects there. As of October 2026 the repository has about 6,200 stars 3.
The unit of work is a task, and a task is a directory: instruction.md, task.toml, an environment/ (a Dockerfile or any spec the runner supports), an optional solution/, and a tests/ folder whose test.sh must write a numeric reward to /logs/verifier/reward.txt or reward.json 1. A trial is one agent attempt at one task; a job is a set of trials. Tasks have no dependency on Harbor itself, so the same directory can be consumed by another runner.
That makes Harbor different from Inspect AI. Inspect is a Python program per eval: you write solvers and scorers. Harbor is a file format plus a runner: you write a Dockerfile and a test script, then point any of its built-in agents at it. If your eval is "did the agent leave the box in the right state," Harbor's shape fits. If it is a multi-sample QA eval with model-graded scoring, it does not.
Figure: Harbor trial mechanism: a task directory (instruction, environment spec, tests) is started in a sandbox such as Docker, Podman, Modal, or Daytona; the chosen agent works inside it; tests/test.sh then writes a numeric reward, and the trial keeps the reward and an agent trajectory. A job fans many trials out in parallel, with -k attempts per task.
Strengths
- Tasks are plain files. A task needs no Python SDK. Anyone who can write a Dockerfile and a shell test can contribute one, which is how Terminal-Bench takes community tasks 2.
- Many agents, many sandboxes, one flag each. The 0.24.0 CLI lists 45 built-in agent entries (claude-code, codex, gemini-cli, openhands, terminus-2, mini-swe-agent, plus
oracle and nop controls) and more than 30 --env values, including docker, podman, kata, modal, daytona, e2b, gke, and vercel. I counted both from harbor run --help and harbor agent list on the installed package. Cloud sandboxes are optional extras, for example harbor[modal] 1.
- Separate verifier and regrade. A task can verify in a fresh sandbox that does not inherit the agent's filesystem changes.
harbor job regrade then re-runs an updated verifier on recorded artifacts without starting the agent, so it needs no agent credentials and costs no agent tokens 1. Fixing a flaky test no longer means paying for the whole run again.
- Oracle runs as a task QA step. The Terminal-Bench README has you run the
oracle agent five times (-k 5) before trusting a sandbox setup, to catch tasks that flake on your infrastructure 2. Most eval tools have no equivalent.
- Trajectories are first-class. Agents emit ATIF, a JSON trajectory format with messages, tool calls, observations, and metrics, and
harbor view browses jobs locally 1. This is the input you need for the trajectory vs outcome analysis.
- Multi-turn trials. v0.22.0 (August 22, 2026) added simulated-user trials over ACP, where a user agent holds the instruction and the agent under test learns the task by conversation; the same release added Vercel Sandbox. v0.23.0 (September 12) added Podman and Kata microVM isolation 4.
harbor run --diff previews what a job would reuse, regrade, or rerun against a prior job 1.
Limitations
- Pre-1.0 and fast. Tagged releases ran from v0.17.0 on July 3 to v0.24.0 on October 5, 2026 4, and the package requires Python 3.12 or newer 5. I tried to install it into a Python 3.11 venv and got a package with no
harbor command. Pin the version in every report, as you would for Inspect AI, and expect CLI flags to move.
- The default verifier shares the agent's sandbox. Unless the task sets
environment_mode = "separate" or gives the verifier its own environment, tests/ is uploaded into the same environment the agent just worked in 1. An agent that edits files the tests read, or the tests themselves, can pass without solving the task. The docs say the separate verifier "improves the security boundary," which is a polite way of saying the default is weaker. Read reward hacking before you publish a number, and treat any task without a separate verifier as suspect.
- Network is open unless the task closes it. The documented
network_mode baseline for the agent environment is public when omitted; no-network or allowlist must be set per task 1. A result earned with web access is a different measurement from one earned offline.
- Agents install at trial time. Many built-in agents are installed into the sandbox when the trial starts, and release notes show recurring fixes to installers for npm packages and curl bootstraps 4. An agent CLI that changes upstream between two runs changes your score. That is environmental drift you did not choose; record the agent version and, where you can, bake it into the image.
- Not a scoring toolkit. Reward is whatever your test script writes. There are no built-in LLM-judge metrics, dataset-level statistics beyond custom metrics, or CI assertions like Promptfoo has. Use
-k attempts and read pass@k and consistency rather than reporting one run.
- Docker is the default and cloud is a bill. Local runs need a container runtime. Terminal-Bench's own guidance runs hundreds of concurrent trials on Modal 2, so budget sandbox hours and model tokens together.
Best fit
Command-line and repository-level agent evals where success is checkable by a script: build the thing, fix the failing test, configure the server. It is the practical way to run Terminal-Bench and to build an internal Terminal-Bench-style suite for your own coding agent. See the autonomous coding task-type page for how these evals fit with SWE-bench-style suites. Strongest fit for research engineers and teams comparing agent scaffolds across models.
A version note: the Harbor docs still show terminal-bench@2.0 in their examples, but the Terminal-Bench site now serves 4.0, a breaking release that removed 8 tasks, updated 19, and set a uniform 8-hour agent timeout, so scores do not carry over from earlier versions and trials must be rerun 6 7. State which dataset version you ran.
Getting started
# Harbor needs Python 3.12 or newer
uv tool install harbor # or: pip install harbor
harbor --version # prints 0.24.0 at the time of review
harbor run --help # every flag below is documented here
The quick start in the docs runs one task in the default Docker sandbox. It needs Docker and a model key, so I did not execute it:
OPENAI_API_KEY="<your-key>" \
harbor run -t hello-world/hello-world \
-a codex -m openai/gpt-5.6-luna
harbor view ./jobs
To run Terminal-Bench 4.0 with five attempts per task on a cloud sandbox, follow the Terminal-Bench README and docs 2 6. --print-config resolves a job without running it, which I ran offline to check the flags parse:
harbor run --print-config \
-d terminal-bench/terminal-bench@4.0.0 \
-a claude-code -m anthropic/claude-sonnet-5 \
-k 5 -n 8 -e daytona
That printed a resolved JobConfig with n_attempts: 5, n_concurrent_trials: 8, environment.type: daytona, and the dataset ref 4.0.0. Drop --print-config and set the provider key (for Daytona, DAYTONA_API_KEY, plus uv tool install "harbor[daytona]") to run it for real 1.
Pricing notes
The framework is free under Apache-2.0 3. You pay for model calls and sandbox time. Harbor Hub offers hosted jobs (harbor run --launch) and result sharing; I did not find a public rate card in the docs reviewed, so check before relying on it. Agent runs on hard tasks are long and token-heavy, so sandbox hours at high concurrency are usually the surprise line item, not the framework.
Alternatives
Inspect AI is the closest alternative and the better choice when you want a Python-defined eval with solvers, scorers, a large prebuilt register, and a mature VS Code viewer; its sandboxing also covers Docker, Kubernetes, and Modal. Harbor wins when your tasks are self-contained directories graded by a script, when you want many off-the-shelf agent CLIs run side by side, or when you need to run Terminal-Bench itself. Inspect also has the larger prebuilt eval register; Harbor has the larger set of ready-made agent adapters. For plain model benchmarks without agents, use lm-evaluation-harness. For CI regression on prompts, use Promptfoo.
Citations and last verified
Verified 2026-10-17 against the Harbor GitHub repository, LICENSE file, PyPI metadata and release notes, the Harbor documentation site, the Terminal-Bench README, and the Terminal-Bench 4.0 announcement. The CLI behavior described here was checked on an installed harbor 0.24.0 in a fresh Python 3.12 virtual environment.