Error analysis is the process of reading actual model outputs, naming the failure modes you see, clustering the names into categories, and prioritizing fixes by frequency and severity. It is the activity with the largest payoff per hour in an eval program. Hamel's NurtureBoss case study found three issues accounted for over 60% of all problems, and date handling, which failed 66% of the time, improved from 33% to 95% success once the team worked from the labeled failures 1.
The three-stage workflow
The pattern inherits from qualitative social science.
Open coding. Read individual traces and write down what went wrong in your own words. No predefined categories. The phrases are the data: "got the date wrong by a day," "confused vegetarian with vegan," "ignored the system message about formality." Start with a pool of 100 diverse traces and annotate at least the first 30 yourself. Note the first failure in each trace, since upstream errors cause the downstream ones; tag every independent failure if you can afford it 2.
Axial coding. Cluster the open-coding phrases into named categories. Clusters emerge from the data, not from a prefabricated taxonomy. The FAQ calls this the most important step of the three 2. "Got the date wrong by a day," "missed the timezone," and "off-by-one on the recurrence" cluster into "date handling."
Structured coding. Re-label the original 100 traces against the now-stable categories. Compute per-category frequency. Order by frequency. The top three typically account for 60 to 80 percent of all problems; in the NurtureBoss case they covered over 60 percent 1.
The whole loop takes a principal expert one or two days the first time and four hours the next. The cadence after that is weekly: ten to twenty fresh production traces, re-label against the existing categories, watch for new categories, prioritize fixes.
Figure: Error analysis for LLM evals in three stages: open coding names each failure in your own words across roughly 100 traces, axial coding clusters the phrases into 3 to 7 categories, and structured coding counts per-category frequency. The top three categories typically cover 60 to 80 percent of all problems, and a weekly loop re-labels 10 to 20 fresh production traces against the category set.
Why bottom-up beats top-down
The reflex pattern is still to organize failures into "hallucination, toxicity, bias, refusal." That taxonomy is too coarse for product-team decisions. "Hallucination" lumps "made up a product feature that does not exist" (catastrophic) with "stated a fact slightly off" (cosmetic).
Bottom-up taxonomies are product-specific. NurtureBoss had a "date handling" bucket that dwarfed every general category. A customer-support team finds "refund policy misquoting." A code agent finds "missed test case." The buckets are operational because the fix is mechanical once named: change the date parsing logic, add the refund policy to the system prompt, add a verifier on missed tests 1.
Top-down taxonomies are useful exactly once: when reporting to a board or a regulator. Not for the team that ships the fix.
The 60-80 percent rule
The working range is three to five named categories covering 60 to 80 percent of all problems; NurtureBoss's top three covered over 60 percent 1. Failure modes are skew-distributed. The prioritization implication is direct: fix the top three, rerun the eval, the aggregate metric moves five to fifteen points, pick the new top three, repeat.
The inverse failure: spreading fixes across ten "important" categories at one percent each, watching the aggregate move a half percentage point per quarter. Chase frequency, not novelty.
Sample-size saturation
The stop rule for the first pass is theoretical saturation: keep reviewing until new traces stop revealing failure modes or changing existing ones. The FAQ asks for at least 100 traces and says to continue past 100 while you are still learning 2. A workable way to apply it is to stop only after about twenty traces in a row produce no new category and no change to an existing one. The twenty is this site's heuristic, not a number from the FAQ.
The first 30 traces produce most of the categories. The next 30 add one or two. The last 40 add zero or one. Total is roughly 100, often shared with the calibration set.
LLM-assisted axial coding
Axial coding is the step LLMs accelerate the most. The FAQ's rule is to open code 30 to 50 traces yourself first, then have an LLM propose groupings from your raw notes, and review and refine the clusters yourself 2. Its current version of the loop goes one step further: after your first 30 annotations an agent clusters the pool and searches the remaining traces for likely instances of the failures you described, and you accept or reject each suggestion until you reach saturation. Keep that first pass manual, because early agent suggestions bias your own judgment 2.
import json
from anthropic import Anthropic
client = Anthropic()
def cluster_failure_modes(open_codes: list[str]) -> list[dict]:
prompt = f"""You are clustering qualitative failure-mode descriptions
from an LLM eval program into 3 to 7 named categories.
Open-coded descriptions:
{json.dumps(open_codes, indent=2)}
Return JSON. Each category has:
- name: a short noun phrase ("date handling", "policy misquoting")
- description: one sentence defining the category
- examples: the original descriptions that map to this category
"""
out = client.messages.create(
model="claude-opus-5-5",
max_tokens=16000,
messages=[{"role": "user", "content": prompt}],
)
text = next(b.text for b in out.content if b.type == "text")
return json.loads(text.strip())
The snippet omits temperature because Claude Opus 4.7 and later return a 400 error for any non-default value 3. It reads the text block by type because Opus 5.5 runs with thinking on and a response can start with thinking blocks, which also count against max_tokens 4. As of 2026-10-01 the Claude docs point most workloads to claude-opus-5-5; claude-opus-4-7 is still active but listed as legacy 3.
The principal expert reviews the proposed categories before they go into structured coding. The pattern is human-in-the-loop, not automated. Three things stay with a human: the initial open coding, validating the taxonomy (an LLM will fold a login crash and a slow login into one "login issues" bucket that needs two different fixes), and ground-truth labels for anything used to validate a judge 2. SPADE-style assertion generation extends the same pattern to assertion synthesis 5.
Principal expert and data viewer
The program needs exactly one principal domain expert. They define the rubric, own the categories, sign off on category changes. Distributing rubric ownership is the failure mode Shankar et al. document: criteria drift compounds when no human has the final say 6. The expert needs deep domain knowledge, has looked at the product's outputs, has two hours a week for this.
Open coding is bottlenecked by reading speed. The single highest-payoff tool an AI team can build in an afternoon is a custom trace viewer with: side-by-side input and output, copy-trace-as-JSON, filter on metadata, annotate with a category tag, a progress counter and keyboard shortcuts, export labels to JSONL 1. Anthropic's eval team makes the same bet: it built tooling for viewing eval transcripts and reads them regularly, because the transcript shows whether a failure is real or the grader is wrong 7. Vendor UIs do not beat a 100-line Streamlit app at this loop because they optimize for observability, not for the expert's reading pattern.
Weekly cadence and failure modes
After the first pass: sample ten to twenty production traces uniformly (or oversample failures if base rate is too low). Label against the existing category set; flag new category candidates. Update the priority list. Spot-check the LLM judge on five labeled traces. The FAQ's version has two tiers: 10 to 20 traces weekly between major analyses, plus a full pass of 100 or more fresh traces each review cycle, typically every two to four weeks and after a new feature, prompt update, or model switch. New systems need weekly analysis until failure patterns stabilize, and mature ones can drop to monthly unless usage changes 2. Shankar's flywheel formalizes the cadence as a database-backed process; the spreadsheet version works for teams of one to three 8.
Three failure modes. Top-down taxonomy creep (team imports OWASP or NIST categories; bottom-up labels stop being product-specific). One-shot labeling (run error analysis once before launch and never again). Solo coding (one engineer labels without expert review). Fixes: keep operational labels separate from reporting labels 1, maintain the weekly cadence, use the principal expert pattern.
What good looks like
A production error-analysis program ships a bottom-up category list (3 to 7 items, owned by a principal expert, versioned with the rubric), a weekly cadence with ten to twenty new production traces labeled, a regression set that grows from analysis output, and a trace viewer that supports the reading pattern. The 60-80 rule and open coding workflow chapters cover the operational scaffolding.