Pairwise comparison asks a rater (human or judge) to pick between two responses on the same query. Aggregated across thousands of pairs, those preferences fit a Bradley-Terry or Elo model and produce a ranked leaderboard with calibrated intervals. The pattern powers Arena (the platform that began as LMSYS Chatbot Arena, ran as LMArena, and in January 2026 renamed itself Arena at arena.ai 1), Arena-Hard-Auto, and AlpacaEval 2.
When pairwise beats absolute scoring
Two cases push toward pairwise. Humans cannot calibrate a scalar on subjective quality; asking five raters for a 1-to-10 score produces five distributions with no shared zero point, while asking which of two is better produces a label every rater reads the same way 2. And ranking many systems against each other without re-labeling each scales better: pairwise is O(n log n) battles with smart sampling; absolute scoring is O(n) per system times the eval-set size.
Pairwise is the wrong tool when a clean binary criterion exists. Direct binary is one judge call per trace, produces a confusion matrix, and forces a falsifiable cutoff. Use pairwise when the rubric is "which feels better"; use direct when the rubric is "is this correct." For agents, pairwise votes are not the only route either: Arena's Agent Arena leaderboard (June 2026) ranks by randomizing component choices in real sessions and estimating causal treatment effects from signals such as confirmed success, steerability, and tool hallucinations, not from pairwise votes 3. Those signal definitions still move: on 2026-09-30 steerability changed from scoring only the fix after a user correction to scoring every assistant turn, so a model that needs fewer corrections now gets credit 4. Re-baseline before comparing agent scores across that date.
Bradley-Terry and Elo
Bradley-Terry models the probability A beats B as a logistic function of a latent skill difference: . Fit by maximum likelihood on the battle log. Elo is the online update form of the same model; each battle nudges both ratings by a step proportional to the gap between the outcome and the predicted win probability. The Arena moved from online Elo to maximum-likelihood Bradley-Terry in December 2023, because the online ratings varied considerably and the intervals built from them were too wide 5. It has since dropped the Elo name: the Arena Score is defined as the Bradley-Terry coefficients, "which we formerly called the Elo score" 6. Ties count as half a win and half a loss in the fit 5.
Sampling has changed more than once. The 2023 launch paired models by prior expectation, then switched to uniform sampling for better coverage of the rankings 7. The 2024 paper went back to non-uniform active sampling, picking each pair in proportion to how much it would shrink the confidence intervals, and corrected for it by inverse-probability weighting in the fit 8. In July 2025 the live board adopted a reweighting that downweights battles inversely to how often they appear, so new and deprecated models are not distorted by uneven exposure 4. The generalization: if a private eval program adaptively picks pairs, the BT fit has to model the sampling design or the ranking drifts.
Figure: The pairwise comparison to Elo pipeline: each battle is judged twice with response positions swapped, agreement records a win and disagreement a tie, and thousands of battles fit a Bradley-Terry model whose leaderboard carries 95% intervals, about plus or minus 3 to 9 points across the top 20 of Arena's text leaderboard as of 2026-10-02.
Confidence intervals on a ranking
A leaderboard without intervals pretends every neighbouring pair is distinguishable. They almost never are. Arena reports an interval on every score. The 2024 paper compared a pivot bootstrap against sandwich robust standard errors in a simulation study and deployed the sandwich intervals 8. Since July 2025 the live board computes them in closed form from the Central Limit Theorem for M-estimators, checked against bootstrap intervals for close parity at a fraction of the compute 4. Across the top 20 of the text leaderboard fetched as of 2026-10-02, intervals run about plus or minus 3 to 4 points for models with 35,000 or more votes and about plus or minus 9 for those under 5,000 votes 9. Differences smaller than the combined interval are noise.
import numpy as np
from scipy.special import expit
from scipy.optimize import minimize
def fit_bt(battles, n_models):
def nll(theta):
full = np.concatenate([theta, [0.0]])
return -sum(np.log(expit(full[w] - full[l]) + 1e-12) for w, l in battles)
out = minimize(nll, x0=np.zeros(n_models - 1), method="L-BFGS-B")
return np.concatenate([out.x, [0.0]])
def bootstrap_bt(battles, n_models, n_boot=1000):
n = len(battles)
samples = np.empty((n_boot, n_models))
for b in range(n_boot):
idx = np.random.randint(0, n, size=n)
samples[b] = fit_bt([battles[i] for i in idx], n_models)
return samples.mean(axis=0), np.percentile(samples, 2.5, axis=0), np.percentile(samples, 97.5, axis=0)
Position bias is non-negotiable
Pairwise judges (LLM or human) favor one slot over the other. Zheng et al. measured first-position preference of 30 percent for GPT-4, 50 percent for GPT-3.5, and 75 percent for Claude v1 on pairs of similar answers, sampled twice from GPT-3.5 10. The standard mitigation: score every pair twice with the order swapped. If both orderings agree, record one win for that response. If they disagree, record a tie. Two judge calls per battle; residual position effect bounded by the disagreement rate. The paper's other option is random slot assignment, which works at scale.
You cannot swap a human vote. When Arena began counting Battles in Direct votes in May 2026, it found a position bias favoring the model on the left and an advantage for models sharing an organization with the prior turns of context. It absorbed both as features in the Bradley-Terry fit (is_direct_battle and same_org_indicator) instead of changing the protocol 4. That regression route is the one to use when a pair cannot be re-run in swapped order.
Length-controlled variants
Verbosity bias stacks on position bias. AlpacaEval introduced length-controlled scoring (LC-AlpacaEval); Dubois et al. report correlation with Chatbot Arena rose from 0.94 to 0.98 after the adjustment 11. If a pairwise program reports raw win rates without controlling for length, the leaderboard is partly measuring which model writes longer.
Arena applies the same idea to human votes. Style control adds answer token length and the counts of markdown headers, bold elements, and lists as extra covariates in the Bradley-Terry regression, so each model coefficient is estimated with style held fixed; length was the dominant factor and the markdown effects were second order 6. On the text leaderboard fetched as of 2026-10-02, Style Control is the selected adjustment by default, beside a Factuality option and a None option 9. The Search leaderboard has defaulted to plain Bradley-Terry since August 2025, while the team studies how style effects differ for search 4. Style control is observational: Arena itself notes unobserved confounders, such as a chain-of-thought explanation that raises both length and quality, so a controlled score is not a style-free score 6.
Preference and correctness are different signals, and Arena now ranks on both. The Factuality adjustment, launched July 2026 for the Text and Search arenas and off by default, audits sampled battles by extracting atomic claims from each response, verifying each with search agents, and letting the response with the higher average claim truthfulness win the audit battle. A composite Bradley-Terry model then blends those labels with human votes at a default factuality weight of 25 percent 12.
| Method | Position control | Verbosity control |
|---|
| MT-Bench | swap pairs | none |
| Arena (human votes) | randomized slot; regression feature for direct battles | style control: length and markdown covariates, selected by default |
| Arena-Hard-Auto | swap pairs | style control, on in the official v2.0 configuration |
| AlpacaEval 2 LC | randomized output order | length-control regression |
How many battles
Sample-size for a Bradley-Terry leaderboard depends on the smallest skill gap you want to separate. Rough order of magnitude: one thousand battles per pair you want to resolve at 95% confidence on a moderate gap. The Chatbot Arena paper ranked models from more than 240,000 votes 8. The live text leaderboard holds 8.6 million votes across 413 models, and the top 20 carry from about 4,400 to 122,000 votes each (as of 2026-10-02) 9. A private eval comparing four prompt versions usually gets clean rank with a few thousand battles total. Arena-Hard-Auto automates the pattern over 500 hard prompts and reports 98.6% correlation with human preference rankings at a cost of about $20 13. Version 2.0 (April 2025) refreshed the prompts, added 250 creative-writing queries, and uses GPT-4.1 or Gemini 2.5 as the judge 14.
New models have no votes yet, so Arena publishes day-one ratings too. AutoEval (July 2026) scores fresh responses with a reward model trained on millions of Arena pairwise votes, turns the score differences into soft proxy votes, and tags the entry AutoEval until enough human votes arrive. With the reward model trained on data through April 2026, AutoEval scores for later releases had a rank correlation above 0.98 with their live scores, and picked the better model in head-to-head tests more than 90 percent of the time once the true gap exceeded 10 points 15. Read a tagged score as a model-predicted preference, not a human one.
What good looks like
A pairwise program ships position swaps on every battle treated as one observation, length control reported alongside raw win rate, bootstrap 95% intervals on every model's score, and a fixed reference model kept across runs as a drift sentinel. The pairwise versus direct chapter covers the operational picks; statistical methods covers the bootstrap.