What "good" looks like
A computer-use or browser-use agent drives a real screen: inputs are screenshots, outputs are keyboard, mouse, and (sometimes) accessibility-tree actions. It is scored by execution-grounded success, meaning a programmatic check confirms the agent actually reached the goal state, not that it looked done.
The primary metric is that success rate on OSWorld (369 single-shot desktop tasks in the original; 108 long-horizon workflows in OSWorld 2.0) or WebArena (self-hosted web domains). The public numbers put the target in context:
- OSWorld, original paper (2024): humans hit 72.36%; the best agent hit 12.24%[1].
- OSWorld, computer-use launch (2024): Anthropic's Claude 3.5 Sonnet reported 14.9% screenshot-only and 22.0% with extra steps[2].
- WebArena, original paper (2023): humans hit 78.24%; GPT-4 hit 14.41%[3].
- OSWorld-Verified today (leaderboard fetched as of 2026-10-01): the top entry is 90.19% at 100 steps (Intelligence-Indeed Agent, an agentic framework that also uses coding-based actions, listed 2026-07-25), and the top general model with no add-ons is 85.96% (claude-fable-5, listed 2026-08-01). Both sit above the roughly 72% human baseline[4].
- WebArena today (community leaderboard sheet, fetched as of 2026-10-01): the top listing is 74.3% (Deepseek v3.2, 02/2026), about four points under the human 78.24%[5].
The original-paper gap has closed. Agents now clear the human baseline on OSWorld-Verified and sit within about four points of it on WebArena, so a 2024 or 2025 paper number is no longer the bar for a current agent. That is the context for the move to long-horizon suites.
The 2026 shift is to long-horizon tasks with partial credit. OSWorld 2.0, announced on the OSWorld site on 2026-06-26, replaced the single-shot desktop tasks with 108 workflows that take a human a median of about 1.6 hours and require an average of 318 tool calls at the frontier, against roughly 30 in the original OSWorld [6]. On its primary binary-completion metric at a 500-step cap, the strongest agent reported completes only 20.6% of tasks, though the partial-credit score reaches 54.8%, and a token-efficient model plateaus near 13% [6]. Two design changes travel with that shift and belong in your own harness: a partial-credit or checkpoint score alongside binary completion, because at 300-plus steps a single binary tells you almost nothing about where the agent stalled; and per-task safety reports that audit safety-sensitive execution separately from task success.
Step budget is a first-class confound here, not a footnote. OSWorld-Verified, the community-fixed release that repaired 300-plus task issues and moved to parallelized cloud execution, documented one model's score swinging from 9.1% to 23.0% purely with the step budget it was given [7]. Report the step cap with every number, and read the pass^k budget section for why a fixed budget both understates and distorts. The Verified leaderboard itself lists runs at caps of 15, 50, and 100 steps, so match the cap before you compare two rows[4].
Figure: Success rates for computer-use and browser-use agents: humans reach 72.4% on OSWorld and 78.2% on WebArena against 12.2% and 14.4% for the best agents in the original papers; OSWorld 2.0 scores 54.8% partial credit but only 20.6% binary completion at a 500-step cap, and OSWorld-Verified saw one model swing from 9.1% to 23.0% on step budget alone.
Production evals do not yet exist as a packaged category here. Most teams roll their own with pytest plus screenshot diffs. Treat the public benchmarks as the floor, not the bar; your domain workflows are the actual eval.
flowchart TD
TASK["Workflow task"] --> AGENT["Agent: screenshots in, mouse/keyboard out"]
AGENT --> CAP{"Actions under the cap?"}
CAP -->|no| LOOP["Kill run: likely a loop"]
CAP -->|yes| STATE{"State-check: goal state reached?"}
STATE -->|no| FAIL["Fail, keep the screen recording"]
STATE -->|yes| SAFE{"Any dangerous action: delete, send-money, mass-email?"}
SAFE -->|yes| REVIEW["Route to safety classifier review"]
SAFE -->|no| PASS["Count as success"]
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Steps to completion | Inefficient or looping runs | Median and 95th percentile actions per successful task |
| Mis-clicks | Wasted or wrong actions | Clicks on coordinates that do not advance the workflow |
| False completion | Agent claims success when the goal state is not reached; the most dangerous mode for destructive workflows | Compare claimed success against the deterministic state-check |
| Screen-state divergence | Drift from the expected end state | Diff against a known target state, when the workflow has one |
Offline eval design
Start with public breadth. OSWorld, WebArena, and Mind2Web for coverage, plus VisualWebArena for multimodal web tasks[8].
Then build the set that predicts whether you ship: an internal "top 20 workflows our users do" set with deterministic completion checks. The check should be programmatic:
- A row appearing in Salesforce with specific fields.
- An email landing in a specific folder.
- A file existing at a specific path.
LaaJ on screenshots is unreliable; deterministic state checks are the only honest score.
Cap actions per run, per workflow. The cap doubles as a cost control and a damage control; an agent that loops indefinitely is both expensive and dangerous. Short internal workflows (a CRM update, a file move) should land in 10 to 30 actions; long-horizon workflows of the OSWorld 2.0 kind need budgets in the hundreds. Whatever you set, report the cap with every number, per the step-budget paragraph above.
Online eval design
Screen-record every run. Storage cost is real (hours of video per day per agent), but the record is the only way to debug failures. Sample-review every Nth failed run.
Alert on two triggers: success-rate drops, and dangerous-action triggers (delete, send-money, mass-email). At launch Anthropic said it had built classifiers that identify when computer use is being used and whether harm is occurring, and told developers to start with low-risk tasks[2]. That is misuse monitoring on the vendor side, not a guard on your agent's destructive clicks. The current docs add a second layer: classifiers scan the screenshots the tool returns for prompt injection and steer the model to check with the user before acting on a flagged instruction, a protection the docs say will not suit every use case, such as ones without a human in the loop[9]. Unless you have opted out, a headless eval measures the agent with that check in its path, so say which configuration you scored. The delete and send-money check is still yours to write.
The hardest part is the action sandbox. With real production credentials, every failed run is a potential incident. With staging credentials, the eval distribution shifts away from production. There is no clean answer; most teams run staging for most evals and reserve production runs for a small monitored sample.
Top benchmarks
- OSWorld 2.0, https://arxiv.org/abs/2606.29537. 108 long-horizon desktop workflows with binary-completion and partial-credit scoring at a 500-step cap, plus per-task safety reports; the current bar for OS-level computer-use[6].
- OSWorld (original) and OSWorld-Verified, https://os-world.github.io/. 369 desktop tasks; the Verified release fixed 300-plus task issues and parallelizes in the cloud, and its leaderboard top is 90.19% as of 2026-10-01[1] [7].
- WebArena, https://arxiv.org/abs/2307.13854. Self-hosted web domains scored by functional correctness; the standard for browser-use[3].
- WebArena-Verified, https://github.com/ServiceNow/webarena-verified. Public release December 4, 2025: all 812 tasks manually re-reviewed, LLM-judge and substring matching replaced by deterministic type-aware scoring, offline scoring from network traces, and a 258-task Hard subset for cheaper runs[10].
- Mind2Web 2, https://arxiv.org/abs/2506.21506. 130 long-horizon agentic-search tasks judged by an Agent-as-a-Judge framework using tree-structured rubrics that score answer correctness and source attribution[11].
- TheAgentCompany, https://arxiv.org/abs/2412.14161. A simulated software company (175 tasks) with an explicit partial-credit formula combining checkpoints and final success[12].
- VisualWebArena, https://arxiv.org/abs/2401.13649. Multimodal extension of WebArena[8].
- OSWorld harness: parity with public scores.
- Browser-Use and Playwright: custom verifiers and screenshot capture.
- Inspect AI: sandboxed runs with strong logging primitives.
- WebArena self-hosted environment: reproducible browser evals. WebArena-Verified installs from PyPI or runs from a Docker image[10].
- Anthropic computer-use tool: on the Claude API and Google Cloud, Claude 5.5 and later models accept computer use only through the
computer_toolset_20260801 toolset and return an error for the earlier computer_20251124 tool, so a harness pinned to the old version cannot score them[9].
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| No deterministic verifiers | LaaJ on screenshots is unreliable | Build state-check functions for every workflow you eval |
| No safety eval | Computer-use can do destructive things | Add your own dangerous-action check and an injection probe. Anthropic's docs list a dedicated VM, a domain allowlist, and human confirmation of consequential actions as precautions you set up yourself[9] |
| Headline OSWorld numbers without the subset | The maintainers allow either all 369 tasks or 361 without the 8 Google Drive tasks, which can fail to initialize on network or credential issues | State which set was scored[4] |
| Conflating "reached the end" with "reached the goal" | A run can finish without hitting the target state | The state-check is the only honest signal |
Canonical reading
Anthropic's computer-use launch post is the most useful single read because it includes both the score on OSWorld and an explicit recommendation about deployment risk[2]. The post calls the capability experimental and imperfect and encourages developers to "begin exploration with low-risk tasks". The OSWorld paper itself is the canonical benchmark reference[1]; its 12.24% is a 2024 number, so pair it with the current Verified leaderboard.
Day-1 starter
- 20 workflows with state-check functions. The state check is the eval; without it, scoring is impressionistic.
- Screenshot logger plus a diff viewer. The viewer is the debugging surface; without it, you cannot understand why a run failed.
- Hard cap on actions per run, budgeted per workflow. Bounds cost and bounds damage. Ten to 30 actions for short internal tasks, hundreds for long-horizon ones; a run far past its own budget is usually a loop.
Cross-cutting links