The argument over synthetic versus human-authored data is mostly miscast. The two are not substitutes; they answer different questions. Human-authored data tells you what your users do and what your team thinks the right answer is. Synthetic data tells you what your system does under inputs you would not otherwise see. A set with only one of them has a blind spot.
The honest accounting
Human-authored examples carry three properties synthetic data cannot fake: they reflect the distribution of inputs your users produce, they carry implicit ground truth from a domain expert who could defend the label, and they anchor the team's mental model of what the product is supposed to do [1]. The cost is real. A principal expert labels ten to thirty examples per hour for a non-trivial product, so a five-hundred-example set is a multi-day commitment.
Synthetic examples flip the accounting. An LLM produces a hundred inputs in two minutes, with systematic coverage of the feature, scenario, persona grid. What you do not get is ground truth: synthesizing the gold answer with the same family of model you are evaluating is the dominant failure mode in synthetic data.
You also do not get frequencies. A failure rate measured on synthetic inputs says nothing about how often users hit that failure in production. The Evals FAQ adds that synthetic inputs miss the structure of specialized documents, are often unrealistic in low-resource languages, and lack subtlety in high-stakes domains such as medicine and law. In those cases use real traces, and compare the synthetic set against real data as soon as you have any [1].
The rule
Synthesize the input. Hand-label the output. The classic failure is using an LLM to generate both the query and the gold answer, then evaluating a second LLM against the first LLM's idea of correctness. The eval measures how similar the two models think, which is high even when both are wrong. Hamel's field guide gives the same instruction: generate user inputs, not expected outputs, so the data does not inherit the biases of the generating model [2].
# synth_inputs.py
from anthropic import Anthropic
client = Anthropic()
def paraphrase(seed: str, n: int = 10) -> list[str]:
prompt = (
f"Rewrite the message below {n} ways. Vary tone, formality, and "
f"information missing. Keep intent identical. Do not invent details "
f"the seed does not provide. One per line.\n\nSeed: {seed}"
)
out = client.messages.create(
model="claude-opus-4-7", max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
)
return [l.strip() for l in out.content[0].text.splitlines() if l.strip()]
The synthetic part buys breadth. The labeling part buys truth.
Figure: Synthetic vs human data for LLM evals: when one LLM generates both the query and the gold answer, the eval scores model agreement, which stays high even when both are wrong. Synthesizing inputs with an LLM (about 100 in 2 minutes) and hand-labeling outputs keeps that breadth anchored to defensible ground truth.
When synthetic ground truth is defensible
Programmatic ground truth is the cleanest exception: a calculator, a SQL executor, a code unit test, or a constraint check produces the gold answer mechanically. Anything that compiles, runs, or matches a regex falls in this bucket. Retrieval is the common case: pick a document from your corpus, extract a fact, and have the LLM write the question that fact answers. The source document is the gold label by construction, and the Evals FAQ recommends this reverse process for building retrieval datasets without manual annotation [1]. It labels retrieval, not answer quality; the generation side still needs human labels. Bootstrap labels also work under an explicit "provisional" flag, with a real backfill date that someone is accountable for. Provisional labels left in for two quarters become canonical by accident.
SPADE and the hybrid pattern
The SPADE paper has an LLM synthesize candidate assertions from a pipeline's history of prompt versions, then selects a minimal set that meets coverage and accuracy requirements, checked against developer-labeled pipeline outputs [3]. Across nine real-world pipelines it cut the number of assertions by 14% and false failures by 21% against simpler baselines. The 2,000-plus pipelines in the abstract is how widely SPADE was deployed inside LangSmith, not the evaluation set. The takeaway is not "synthetic assertions work"; it is "synthetic-propose, labeled-data-dispose works." The LLM produces candidates at scale, a small human-labeled sample decides which ones survive, and labeling a sample is cheaper than writing the assertions.
Newer tooling follows the same split. Anthropic's open-source Bloom, released in December 2025, has an agent generate behavioral-eval scenarios and a judge model score the transcripts. The authors hand-labeled 40 transcripts to check the judges, and the strongest one (Claude Opus 4.1) reached a Spearman correlation of 0.86 with the human scores [4]. Generation is automated; trust still rests on a small human-labeled sample.
CAUTION
The propose-and-dispose loop only works if the disposal step can actually reject. If the labeled sample is too small or too clean to expose a bad candidate, or the workflow nudges acceptance (defaults to accept, accepts on enter, hides input details), low-quality candidates slip in and the set quietly degrades. Shankar flags the same risk for LLM-prefilled labels that humans may edit: it is unclear how well the labeler stays aligned with human judgment over time if people do not verify its labels [5].
A target ratio
For a mature dataset a workable starting mix is roughly half human-sampled from production, a quarter synthetic with hand-labels, and a quarter adversarial cases the team wrote to lock down known bugs and policy boundaries. This is a rule of thumb from practice, not a measured result; neither the field guide nor the Evals FAQ prescribes a split. The exact numbers move with the product, but the principle holds: production anchors the distribution [5], synthetic fills coverage gaps, and adversarial cases catch regressions.
Three diagnostics expose a bad mix. Pass rates that disagree by more than ten points between subsets mean the synthetic part is not probing the same surface as the human part. Failure modes that exist only in synthetic examples mean the synthesizer is generating inputs production never sees. Failure modes that exist only in human examples mean synthetic coverage is missing the cells that break. The fix in each case is to pull fresh production traces, label them, and rebalance.