Classical ML evaluation assumes a fixed model, a fixed input shape, a deterministic forward pass, and ground truth two annotators would agree on. LLM evaluation breaks every one of those assumptions. This page names the breaks specifically so the rest of the site can refer back to them.
Non-determinism
Two runs of the same prompt rarely produce the same answer. Even at temperature zero, hosted endpoints are not reproducible. Floating-point addition is not associative, so the order in which a GPU kernel reduces numbers changes the result, and common kernels reduce in a different order at different batch sizes. Server load sets the batch size, so the same request meets different numerics from one call to the next [1]. In Thinking Machines' test, 1,000 temperature-zero completions of one prompt on an open-weights Qwen3-235B model produced 80 unique outputs, and the first divergence came at token 103. The output is a sample from a distribution, not a fixed mapping.
Newer APIs remove the lever. As of October 2026, Claude Opus 4.7 and later models reject any non-default temperature, top_p, or top_k with a 400 error, and Anthropic's migration guide notes that temperature 0 never guaranteed identical outputs on earlier models [2]. Design the eval for sampling variance instead of trying to switch it off.
A single eval run is one realization of a noisy process. A 2-point movement on a 100-example set is usually inside the run-to-run variance of the same model on the same prompt. The fix is to treat eval runs as samples with confidence intervals, paired comparisons, and replications, rather than as point measurements. Resample each question several times and score the question-level mean; Miller's error-bars paper recommends this to shrink within-question variance [3].
Subjectivity and criteria drift
"Helpful" is not a property two annotators agree on without a shared rubric. The harder finding from Shankar et al. is that the rubric itself only becomes visible once the team has read outputs. Criteria emerge from the act of grading and keep moving as the grader sees more examples [4]. They call this criteria drift and document it across teams who thought they had a stable rubric.
The rubric is a downstream artifact of error analysis, not an upstream input. Teams who try to write the rubric first, then label data, end up rewriting it after the first hundred labels. Treat rubric revisions as version-controlled changes that invalidate prior labels until the team relabels.
Prompt sensitivity
LLM accuracy on a fixed task swings with surface-level prompt changes humans would treat as equivalent. Sclar et al. showed up to 76 accuracy-point swings on LLaMA-2-13B from prompt-format perturbations that preserved the question's meaning (different separators, option-label styles, whitespace) [5]. The effect persists across model scales and families, and the rank ordering of models changes with the prompt format.
Benchmark rankings are a function of the benchmark plus the prompt-format choice. Reporting a single number per model without the prompt format is reporting a fraction of the experiment.
The Anthropic team flagged a smaller related effect: switching multiple-choice option labels from (A) ... (D) to (1) ... (4) shifts MMLU scores by roughly five percentage points [6]. Five points is the size of a generation-over-generation improvement vendor blogs announce as a leap.
If a five-point swing is downstream of a label-format choice, a five-point movement on a public benchmark could be a model improvement or a format change in the eval rig. Without the rig code pinned, the number is not interpretable.
(A) Paris (1) Paris
(B) London (2) London
(C) Berlin (3) Berlin
(D) Madrid (4) Madrid
Same question. Different number.
Figure: Why LLM benchmark numbers are hard to interpret: prompt-format perturbations swing LLaMA-2-13B accuracy up to 76 points and switching MMLU option labels from (A) to (1) shifts scores about 5 points, the size of a typical generation-over-generation gain. Run-to-run noise alone spans about 2 points.
Multi-dimensional quality
A response can be factually correct and badly written, well-written and miss the question, or answer the question and ignore the policy disclaimer. Classical ML metrics (accuracy, F1, ROC-AUC) collapse to a single scalar; LLM quality has at least three independent axes (correctness, instruction-following, safety) and often more for a given product [7]. Aggregating them into one score hides the failure mode you care about.
Score each axis separately and decide release gates per axis. A 2-point bump in correctness that comes with a 4-point regression in instruction-following is not a win. A scorecard that hides the trade-off ships the regression.
Why classical ML frameworks fail
Three assumptions break. The test set is supposed to be a fixed sample from the production distribution; for LLMs the production distribution drifts and the model itself changes under provider revisions. Ground truth is supposed to be a label two annotators agree on; for open-ended generation the label is a rubric output and rubrics drift [4]. Public benchmarks are supposed to be uncontaminated; by 2026 the major ones are contaminated, and MMLU-Redux found 6.5 percent of MMLU items have label errors or ambiguity [8].
Binary judgments with written critiques collapse the agreement problem to a coarser but more stable signal [9]. None of the breaks alone is fatal; together they require a different discipline, which the rest of the site builds.