A quality scorecard with no cost column kills projects six months in when finance discovers the per-request line item. A quality scorecard with no latency column ships a model that wins offline and falls over under p95 traffic. Cost and latency are not infrastructure concerns separate from the eval program. They are eval metrics on the same scorecard, sampled from the same traces, reported in the same table.
The opinion: every release scorecard has three columns at minimum, beyond per-eval quality scores. Median latency, p95 latency, cost per request. Every comparison is reported as a quality-cost-latency triple with intervals on each, and the release decision is "which corner of the Pareto frontier does the product need" rather than "which is best." Huyen's GenAI platform piece is the cleanest statement of why these three live together in production 1.
Why a single quality number is not enough
Three patterns recur in production releases reliably.
The first is the model that wins quality by spending more tokens. Verbose responses, extra tool calls, longer CoT. The quality column lights up; the cost column doubles; the user complaint queue fills with "too long." LLM judges are known to prefer longer answers: Zheng et al. list verbosity among the biases of LLM-as-a-judge, and AlpacaEval added a length-controlled win rate because it favored models that generate longer outputs 2 3. A single-axis judge can score the extra tokens as quality.
The second is the model that wins quality and triples p95. The median moves a little, the tail moves a lot, and the SRE on-call discovers the change Monday morning when the dashboard goes red. Latency at p95 is not a finance concern; it is a UX concern that finance will inherit when retention slips.
The third is the model that hits the quality bar in offline and misses it in online because the offline judge does not penalise the costly path. The fix is not "judge harder"; it is "score the same trace on three axes."
The minimum scorecard
Four columns per release, one row per slice.
- Quality (the primary eval metric, with its Wilson interval).
- p50 latency in milliseconds (with a bootstrap interval).
- p95 latency in milliseconds (with a bootstrap interval).
- Cost per request in dollars, broken into model cost and tool cost (with a bootstrap interval).
On streaming surfaces, add time to first token as its own column. Huyen lists it separately from time per output token and total latency, and vendor speed features can move one without the others 1 4.
Add a row per slice (intent, persona, surface). A scorecard that reports only the global mean hides the slice where one column moved badly. The same applies to release comparisons: candidate vs baseline, per slice, all three columns.
Sampling for cost and latency is different from sampling for quality. Latency and cost are per-trace logged on every request, so the dashboard runs over the full population, not the judged sample. The interval on p95 needs a bootstrap or a quantile-specific formula because the normal approximation does not apply at the tail. Miller's error-bar formulas cover means and proportions of eval scores, where the central limit theorem makes a bootstrap unnecessary; a tail quantile of latency is not that case 5.
Computing the columns
"""
Compute the four-column scorecard with intervals. Quality is from the
judge sample; latency and cost are from the full trace stream.
"""
import numpy as np
from statsmodels.stats.proportion import proportion_confint
def wilson(successes: int, n: int, alpha: float = 0.05):
return proportion_confint(successes, n, alpha=alpha, method="wilson")
def bootstrap_quantile(values, q: float, n_boot: int = 5000, alpha: float = 0.05, rng=None):
rng = rng or np.random.default_rng(0)
arr = np.asarray(values)
boots = np.array([
np.quantile(rng.choice(arr, size=len(arr), replace=True), q)
for _ in range(n_boot)
])
return float(np.quantile(boots, alpha / 2)), float(np.quantile(boots, 1 - alpha / 2))
def bootstrap_mean(values, n_boot: int = 5000, alpha: float = 0.05, rng=None):
rng = rng or np.random.default_rng(0)
arr = np.asarray(values)
boots = np.array([
rng.choice(arr, size=len(arr), replace=True).mean()
for _ in range(n_boot)
])
return float(np.quantile(boots, alpha / 2)), float(np.quantile(boots, 1 - alpha / 2))
def scorecard_row(name: str, judged: list[dict], traces: list[dict]) -> dict:
successes = sum(1 for t in judged if t["pass"])
n = len(judged)
lat_ms = [t["latency_ms"] for t in traces]
cost_usd = [t["cost_usd"] for t in traces]
return {
"slice": name,
"quality_rate": successes / n,
"quality_ci": wilson(successes, n),
"p50_latency_ms": float(np.median(lat_ms)),
"p50_ci": bootstrap_quantile(lat_ms, 0.5),
"p95_latency_ms": float(np.quantile(lat_ms, 0.95)),
"p95_ci": bootstrap_quantile(lat_ms, 0.95),
"cost_per_request_usd": float(np.mean(cost_usd)),
"cost_ci": bootstrap_mean(cost_usd),
}
Three reporting habits worth adopting. First, report cost in dollars, not tokens; tokens are a number engineers argue about, dollars are the number leadership reads. Second, break cost into model and tool components on agent workflows; a quality win that triples tool-call cost without moving model cost is a different story from one that doubles model cost. Third, date the rate card and compute dollars from the billed usage fields on each trace, because cost per request moves while your system stands still.
As of October 2026 (fetched 2026-10-01), Anthropic lists Claude Sonnet 5.5 at 10 per million output tokens, Opus 5.5 at 20, and Haiku 4.5 at 5. Cache reads cost 0.1x the input price (0.05x on Opus 5.5), and the Batch API takes 50% off input and output 6. OpenAI lists gpt-6.1-sol at 0.10 cached input, and 2 and 3 and $15 on 2026-09-01, and the pricing page now says that increase will not occur 6. Tokenizers move the other half of the product. Anthropic states that Claude 4.7 and later models use a newer tokenizer that produces roughly 30% more tokens for the same text, so a model upgrade at an unchanged per-token price can still raise cost per request 6. A scorecard that compares token counts misses this; one that compares dollars per request does not.
Reasoning adds a second source of variance. On Claude models with adaptive thinking, thinking tokens are billed as output tokens, and usage.output_tokens_details.thinking_tokens reports how many of the billed output tokens were reasoning 7. Log it as its own column on reasoning workloads, and make the effort setting part of the row key. Effort moves latency and cost (the default is medium on Opus 5.5 and high on most other models), and changing it between requests invalidates the prompt cache, so a candidate has to be scored at the effort you will ship 7. The OpenTelemetry GenAI conventions, still at Development status, define gen_ai.usage.reasoning.output_tokens and gen_ai.usage.cache_read.input_tokens, so both splits can come from standard spans 8.
The Pareto framing
Once three axes are on the table, "best" stops being a useful question. The right question is "where on the quality-cost-latency frontier should this product live." Customer support: high quality, latency over cost. Bulk summarisation: cost over latency, quality at the floor. Internal coding assistant: latency over quality and cost in that order.
The release scorecard should make the frontier visible. For two candidates with similar quality, plot p95 latency against cost per request, and pick the one closer to the product's corner of the frontier. The single-number-with-CI comparison is still the unit of decision, but the frontier is the context the decision is made in 9.
Vendors now sell points on the frontier directly. As of 2026-10-01, Anthropic's fast mode, a research preview on Opus 5.5, Opus 5, and Opus 4.8, delivers up to 2.5x output tokens per second from the same model, aimed at generation speed rather than time to first token. It prices Opus 5.5 at 40 output per million tokens against 20 at standard speed, and switching speeds misses the prompt cache 4 6. OpenAI's Fast tier, renamed from Priority processing on 2026-07-30, lists gpt-6.1-sol at 20.00 output against 10.00 standard 10. At the other corner, batch tiers halve the price for asynchronous delivery: Anthropic's Batch API is 50% off, and OpenAI's Batch tier lists gpt-6.1-sol at 5.00 6 10. Score each tier as its own candidate row. A speed tier changes latency and cost together and, per Anthropic, leaves the model weights alone, but confirm the quality column anyway.
Figure: Quality-cost-latency Pareto frontier for an LLM release decision: candidates A (p95 820 ms, $0.014 per request) and B (p95 2400 ms, $0.006 per request) tie on quality and sit on the frontier, while C (2600 ms, $0.014) is dominated on both axes. Latency-first products like support and coding assistants pick the low-latency corner; bulk summarisation picks the low-cost corner.
Alert thresholds
Three thresholds worth committing to as defaults.
- Quality regression: 95% interval on the delta excludes zero on the negative side, evaluated per slice 11.
- Latency regression: p95 delta exceeds 20% of the baseline, sustained over a one-hour window. Spikes shorter than that are usually infrastructure noise.
- Cost regression: cost per request delta exceeds 15% of the baseline, sustained over a one-day window. Shorter windows alarm on traffic mix shifts.
A scorecard with alarms on all three axes catches the three failure modes above at the time they ship, not three weeks later when the finance review surfaces them.
TIP
A release that wins quality and loses cost or latency at the configured threshold is a regression candidate. The default is to revert and re-scope, not to ship the quality win and absorb the loss silently. Moshkovich et al. frame agentic optimization the same way: SLAs set quality, latency, and cost requirements, and fluctuations in all three have to be balanced to meet them 12.
What to do this week
- Add three columns to your release scorecard: p50, p95, cost per request. Pull them from the same trace stream that already carries the quality scores.
- Convert at least one ongoing model comparison into a quality-cost-latency triple. Compute intervals on each.
- Set the alert thresholds above. Wire them to the same channel as the quality drift alarms; the runbook is the same.