What "good" looks like
A computer-use or browser-use agent drives a real screen: inputs are screenshots, outputs are keyboard, mouse, and (sometimes) accessibility-tree actions. It is scored by execution-grounded success, meaning a programmatic check confirms the agent actually reached the goal state, not that it looked done.
The primary metric is that success rate on OSWorld (369 single-shot desktop tasks in the original; 108 long-horizon workflows in OSWorld 2.0) or WebArena (self-hosted web domains). The public numbers put the target in context:
- OSWorld: humans hit 72.36%; the best agent in the original paper hit 12.24%[1].
- OSWorld, computer-use launch: Anthropic's Claude 3.5 Sonnet reported 14.9% screenshot-only and 22.0% with extra steps[2].
- WebArena: humans hit 78.24%; GPT-4 hit 14.41% in the original paper[3].
The headline gap between human and agent is large enough that the category is still genuinely nascent.
The 2026 shift is to long-horizon tasks with partial credit. OSWorld 2.0 replaced the single-shot desktop tasks with 108 workflows that take a human a median of about 1.6 hours and require an average of 318 tool calls at the frontier, against roughly 30 in the original OSWorld [4]. On its primary binary-completion metric at a 500-step cap, the strongest agent reported completes only 20.6% of tasks, though the partial-credit score reaches 54.8%, and a token-efficient model plateaus near 13% [4]. Two design changes travel with that shift and belong in your own harness: a partial-credit or checkpoint score alongside binary completion, because at 300-plus steps a single binary tells you almost nothing about where the agent stalled; and per-task safety reports that audit safety-sensitive execution separately from task success.
Step budget is a first-class confound here, not a footnote. OSWorld-Verified, the community-fixed release that repaired 300-plus task issues and moved to parallelized cloud execution, documented one model's score swinging from 9.1% to 23.0% purely with the step budget it was given [5]. Report the step cap with every number, and read the pass^k budget section for why a fixed budget both understates and distorts.
Figure: Success rates for computer-use and browser-use agents: humans reach 72.4% on OSWorld and 78.2% on WebArena against 12.2% and 14.4% for the best agents in the original papers; OSWorld 2.0 scores 54.8% partial credit but only 20.6% binary completion at a 500-step cap, and OSWorld-Verified saw one model swing from 9.1% to 23.0% on step budget alone.
Production evals do not yet exist as a packaged category here. Most teams roll their own with pytest plus screenshot diffs. Treat the public benchmarks as the floor, not the bar; your domain workflows are the actual eval.
flowchart TD
TASK["Workflow task"] --> AGENT["Agent: screenshots in, mouse/keyboard out"]
AGENT --> CAP{"Actions under the cap?"}
CAP -->|no| LOOP["Kill run: likely a loop"]
CAP -->|yes| STATE{"State-check: goal state reached?"}
STATE -->|no| FAIL["Fail, keep the screen recording"]
STATE -->|yes| SAFE{"Any dangerous action: delete, send-money, mass-email?"}
SAFE -->|yes| REVIEW["Route to safety classifier review"]
SAFE -->|no| PASS["Count as success"]
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Steps to completion | Inefficient or looping runs | Median and 95th percentile actions per successful task |
| Mis-clicks | Wasted or wrong actions | Clicks on coordinates that do not advance the workflow |
| False completion | Agent claims success when the goal state is not reached; the most dangerous mode for destructive workflows | Compare claimed success against the deterministic state-check |
| Screen-state divergence | Drift from the expected end state | Diff against a known target state, when the workflow has one |
Offline eval design
Start with public breadth. OSWorld, WebArena, and Mind2Web for coverage, plus VisualWebArena for multimodal web tasks[6].
Then build the set that predicts whether you ship: an internal "top 20 workflows our users do" set with deterministic completion checks. The check should be programmatic:
- A row appearing in Salesforce with specific fields.
- An email landing in a specific folder.
- A file existing at a specific path.
LaaJ on screenshots is unreliable; deterministic state checks are the only honest score.
Cap actions per run, per workflow. The cap doubles as a cost control and a damage control; an agent that loops indefinitely is both expensive and dangerous. Short internal workflows (a CRM update, a file move) should land in 10 to 30 actions; long-horizon workflows of the OSWorld 2.0 kind need budgets in the hundreds. Whatever you set, report the cap with every number, per the step-budget paragraph above.
Online eval design
Screen-record every run. Storage cost is real (hours of video per day per agent), but the record is the only way to debug failures. Sample-review every Nth failed run.
Alert on two triggers: success-rate drops, and dangerous-action triggers (delete, send-money, mass-email). Anthropic shipped new classifiers specifically for the dangerous-action category at computer-use launch[2]; the recommendation for "lower-risk applications" is explicit.
The hardest part is the action sandbox. With real production credentials, every failed run is a potential incident. With staging credentials, the eval distribution shifts away from production. There is no clean answer; most teams run staging for most evals and reserve production runs for a small monitored sample.
Top benchmarks
- OSWorld harness: parity with public scores.
- Browser-Use and Playwright: custom verifiers and screenshot capture.
- Inspect AI: sandboxed runs with strong logging primitives.
- WebArena self-hosted environment: reproducible browser evals.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| No deterministic verifiers | LaaJ on screenshots is unreliable | Build state-check functions for every workflow you eval |
| No safety eval | Computer-use can do destructive things | Add safety classifiers; Anthropic shipped new ones at launch for exactly this reason[2] |
| Headline OSWorld numbers without the subset | Some teams exclude the 8 Google Drive tasks because credential issues make them flaky | State which of the 369 tasks were scored |
| Conflating "reached the end" with "reached the goal" | A run can finish without hitting the target state | The state-check is the only honest signal |
Canonical reading
Anthropic's computer-use launch post is the most useful single read because it includes both the score on OSWorld and an explicit recommendation about deployment risk[2]. The line that matters: capability "remains experimental and imperfect"; recommend "lower-risk applications". The OSWorld paper itself is the canonical benchmark reference[1].
Day-1 starter
- 20 workflows with state-check functions. The state check is the eval; without it, scoring is impressionistic.
- Screenshot logger plus a diff viewer. The viewer is the debugging surface; without it, you cannot understand why a run failed.
- Hard cap on actions per run, budgeted per workflow. Bounds cost and bounds damage. Ten to 30 actions for short internal tasks, hundreds for long-horizon ones; a run far past its own budget is usually a loop.
Cross-cutting links