The cheapest way to mislead yourself with an eval set is to seed it with inputs that look like the demo. The demo path is the one the team practiced; the failures live elsewhere. Dimensional sampling is the discipline of constructing inputs that probe the corners on purpose, by enumerating axes that matter for the product and generating combinations across them [1].
The three-axis frame
Hamel's recipe is features by scenarios by personas. A feature is a capability the product offers (booking lookup, document Q+A, refund flow). A scenario is a state of the world the feature has to handle (happy path, no results, ambiguous query, conflicting documents, irate user). A persona is who is asking (first-time visitor, power user, admin, attacker). Cover the cells of that grid and the eval set probes the product instead of confirming the demo [2].
The three axes are not sacred. Some products need four (add channel: voice, SMS, chat, email). Some collapse to two. Pick the axes whose interactions you suspect will surface failures, not the labels in the original recipe.
Figure: Dimensional sampling grid for synthetic eval test data: 4 features x 5 scenarios x 4 personas = 80 cells, and sampling 1-2 inputs per plausible cell yields a roughly 100-input seed set that probes the failure surface instead of the demo path.
Skip dimensional sampling and pull a thousand production traces at uniform random and two things go wrong.
Production traffic is skewed toward a few common request types. A random sample is dominated by the head, and the head is mostly working. The failures live in the sparse tail, and a small random batch can miss rare cases [1]. A targeted sample built from the three-axis grid hits the tail on purpose.
Some scenarios are rare in production but high-severity. "User is hostile and the model refuses incorrectly" may show up in two out of a thousand traces, but a wrong response is a policy violation. Dimensional sampling forces the team to write that case in even when production has not surfaced enough of it. That matches how Hamel's FAQ says to use synthetic data: to start error analysis before you have traffic, or to test a known failure that appears rarely in real data [1].
Random is still worth keeping. The FAQ keeps some random traces in every review batch, because they can surface failure modes that none of your current signals describe [1]. The grid and the random pull answer different questions.
A working example
The grid for a recipe assistant might look like this.
DIMENSIONS = {
"feature": [
"recipe-lookup",
"substitution-question",
"dietary-conversion",
"multi-turn-refinement",
],
"scenario": [
"happy-path",
"ambiguous-query",
"conflicting-constraints",
"out-of-scope-request",
"user-corrects-bot",
],
"persona": [
"first-time-visitor",
"power-user",
"dietary-restricted",
"hostile-tester",
],
}
# 4 x 5 x 4 = 80 cells. Sample 1 to 2 per cell for ~100 inputs.
import itertools
seed_set = [
{"feature": f, "scenario": s, "persona": p}
for f, s, p in itertools.product(*DIMENSIONS.values())
]
The grid does not need to be fully populated. Pick the cells whose interaction is most likely to break the product (hostile tester plus out-of-scope-request plus refund flow) and skip the implausible combinations (admin plus happy-path plus dietary-conversion). Commit to which cells the team covered and which they did not [2].
The code above is the cross-product route: enumerate every combination, then filter out the implausible ones. It guarantees coverage of the corners and fits when most combinations are valid. The alternative is to have an LLM propose tuples directly, which reads as more realistic but drifts generic and misses rare scenarios; use it when most combinations are invalid [1]. Either way, write about twenty tuples by hand first, so you learn the shape of the problem space before a model scales it [1].
Convert cells to natural language
The grid is a tuple of labels; the eval input is a sentence. Use an LLM to do the conversion in a separate step from tuple generation, then have a human edit for realism. Generating tuples and queries in one prompt gives repetitive phrasing [1]. The LLM produces fluent variation; it is mediocre at writing the way a real annoyed customer writes. The human pass tightens phrasing and catches generic outputs.
A single conversion prompt suffices: "Convert this tuple into one realistic user message. Vary tone. Do not name the dimensions. Be specific." Sample one or two outputs per cell and hand-edit. If the model accepts a temperature setting, raise it for variety; Claude 4.7 and later models return a 400 error on a non-default temperature (as of October 2026), so with those, vary the persona and tone in the prompt instead [3]. The full hundred-input seed set takes a senior engineer plus a domain expert about half a day on the first pass.
Then check that each input triggers its cell. A "no results" cell is only a no-results test if the query returns zero results against your data, and a scheduling cell needs real availability windows [2]. Run the inputs through the full system and review the resulting traces; a pool of roughly 100 diverse traces is a usual starting point for failure discovery [1]. The generate-synthetic-data skill in the evals-skills plugin that Hamel and Shreya maintain walks a coding agent through this same dimensions, tuples, queries, traces flow [4].
When to grow the grid
As error analysis surfaces failure modes the grid did not anticipate, add new axes. If the bot fails systematically on Spanish-language inputs, language is an axis. If failures concentrate around midnight requests, time-of-day is an axis. The grid evolves with the product [5].
The signal that the grid is spent is that error analysis on its inputs stops surfacing new failure modes, which qualitative researchers call theoretical saturation: new reviews stop revealing failure modes or changing existing ones [1]. If three consecutive open-coding passes find the same modes, the grid has saturated the input space. Expand by adding an axis the team has been ignoring, not by adding more examples per existing cell.
What dimensional sampling does not do
It does not replace open coding on production traces. The grid covers the inputs the team can anticipate; production covers the ones the team did not [6]. The right setup is a grid-built seed set for offline regression plus a rolling sample of production traces for ongoing error analysis.
It does not tell you how common a failure is. The grid weights cells by what you chose to cover, not by traffic, so a failure rate measured on synthetic inputs says little about the rate in production [1]. It is also the wrong starting point once you have a hundred or more representative real traces: review those first and spend synthetic inputs on the gaps [4]. And it misleads in some domains. LLM-written inputs miss the structure of specialized documents such as legal filings or medical records, read as unrealistic for low-resource languages and dialects, and lack the edge cases of high-stakes domains, so seed those grids from real examples [1].
It does not pick the test set for a model evaluation. The grid is for the stage-1 and stage-2 evals the team builds and grades against. Picking the model itself against a frontier benchmark is a different problem; see the datasets section.