What "good" looks like
The primary metric is a composite of three signals. No single one is sufficient:
- Human preference: pairwise A/B against the prior model or a strong baseline. The truth, but slow and expensive.
- Prompt adherence: CLIP score, computed with the CLIP model from Radford et al.[1]. Rewards text alignment, not aesthetic quality. Its text encoder behaves like a bag of words, so it cannot tell "the horse is eating the grass" from the reverse. Add VQAScore, which asks a VQA model whether the image shows the prompt, for relations and counting[2].
- Realism: FID (Frechet Inception Distance), from Heusel et al.[3]. Rewards realism, not whether the image matches the prompt.
The target depends on the use case. Stock-photo replacement needs FID below domain thresholds and a CLIP score that beats your prior model on a fixed prompt set. Creative tools need human preference above 50% versus the baseline.
Always report all three. Reporting only FID rewards realistic but off-prompt images.
flowchart TD
GEN["Generated image"] --> CLIP["CLIP score beats prior model?"]
GEN --> FID["FID below domain threshold?"]
GEN --> PREF["Human preference above 50%?"]
GEN --> SAFE{"NSFW classifier pass?"}
SAFE -->|no| BLOCK["Block, regression signal"]
SAFE -->|yes| GATE{"All three signals clear?"}
CLIP --> GATE
FID --> GATE
PREF --> GATE
GATE -->|yes| SHIP["Ship"]
GATE -->|no| FIX["Fix weak axis, never FID alone"]
Figure: Coverage matrix for image generation evaluation: human preference covers prompt match and realism but is slow and costly, CLIP score measures prompt adherence but is blind to realism, and FID measures realism but is blind to prompt match. No single signal covers every column, which is why the primary metric is the composite of all three.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Compositionality | Counting (three cats), spatial relations (cat on top of dog), color binding (red ball next to blue cube) | Per-property pass/fail vs the prompt |
| Text-in-image fidelity | Legible specified text rendered | Rendered text matches the requested string |
| Safety | NSFW flags, IP concerns (real faces, trademarked logos), missing provenance mark where the law requires one[4] | Automated classifier plus IP checks plus a mark-present check |
| Aesthetic preference | Visual quality, decoupled from prompt match | Own rubric, not coupled to prompt adherence |
| Prompt-leak | Model renders prompt text into the image (an extreme failure mode) | Flag images drawing the prompt as text |
Offline eval design
Start on the public sets. PartiPrompts is the canonical creative-prompt set: 1,600+ prompts across categories from Google Research[5]. HEIM (Holistic Evaluation of Text-To-Image Models) gives cross-axis comparison[6].
Use a compositional set for adherence. GenEval 2 has 800 prompts scored with Soft-TIFA, and it exists because the original GenEval drifted from human judgment, with absolute error as high as 17.7% on current models[7]. GenAI-Bench has 1,600 compositional prompts with over 15,000 human ratings[2]. If you still report the original GenEval, spot-check it against raters.
Shortlist with public arena Elo. Before spending rater time, read the public pairwise leaderboards. As of 24 September 2026 (fetched as of 2026-10-02), the arena.ai text-to-image board (formerly LMArena, 6.48 million votes, 80 models) has gpt-image-2.5-sunburst first at 1424, with all of the top eight proprietary[8]. Arena voters judge their own prompts, not yours, so treat the board as a shortlist and keep your preference set as the verdict.
Add an in-house set. Build a creative-brief set of 50 to 200 prompts from your product distribution, paired with 3-rater pairwise preference labels against the baseline. Three metrics to track: CLIP score per prompt, FID over batches, and the pairwise preference.
Run an NSFW classifier on every generation. The classifier-flag rate is a leading indicator of distribution drift after model swaps.
Check the EU dates if you ship there. Article 50 transparency rules have applied since 2 August 2026. Providers of generative systems already on the market before that date have until 2 December 2026 to meet the machine-readable marking duty in Article 50(2), and the new prohibition on systems that generate non-consensual sexual deepfakes and child sexual abuse material also starts on 2 December 2026[4]. Add a check that every output carries the mark your provider says it does, and keep the classifier pass in the release gate.
Online eval design
Implicit feedback is the primary online signal. Saves, edits, and re-rolls carry the most information. A high re-roll rate on a fixed prompt distribution is the leading indicator of quality regression. Layer explicit thumbs-up and thumbs-down per image on top.
Keep the safety pass live. Run the NSFW classifier on every output. Alert on classifier-flag-rate increases and on style-drift signatures after model swaps: a swap can shift the color palette or composition style, and users notice.
Top benchmarks
- PartiPrompts, https://github.com/google-research/parti. 1,600+ creative prompts; the standard for creative-prompt breadth[5].
- HEIM, https://arxiv.org/abs/2311.04287. Holistic eval across 12 aspects, including text-image alignment, quality, aesthetics, originality, bias, and toxicity, over 62 scenarios and 26 models at publication[6].
- GenEval 2, https://arxiv.org/abs/2512.16853. 800 compositional prompts scored with Soft-TIFA, built to resist the drift that hit the original GenEval[7].
- GenAI-Bench, https://arxiv.org/abs/2404.01291. 1,600 compositional prompts with human ratings, paired with VQAScore[2].
- Your in-house creative-brief set. 50 to 200 prompts matching your product distribution; the eval that predicts shipping decisions.
- torchmetrics: FID and CLIP score implementations.
- PartiPrompts harness: prompt-set parity across runs.
- Internal pairwise rating tool: most teams build their own; the simplest shows two images side by side and records the click.
- LangSmith or Braintrust: integrate ratings with the rest of your eval stack.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| FID-only | FID correlates poorly with text-image alignment; a model can have great FID and ignore the prompt | Use the CLIP-plus-FID-plus-preference composite; no one of the three is enough |
| Tiny eval set | A 20-prompt eval has too much variance to detect anything but enormous changes | Use PartiPrompts scale (1,600 prompts for a reason) |
| No safety eval | NSFW and IP flags are production-critical; without a classifier-pass, regression detection here is impossible | Run a classifier pass on every output |
| Not looking at outputs | Failure modes (extra fingers, prompt drift) show by eye but no metric surfaces them | Spend 30 minutes inspecting outputs, as Hamel's field guide argues[9] |
Canonical reading
The FID paper is the canonical reference for the realism metric because it formalizes the distribution-distance interpretation that underpins most modern image quality work[3]. The CLIP paper is the canonical reference for the alignment metric[1]. Read them together: FID for what a "good" image looks like in distribution, CLIP for whether it matches what was asked.
Day-1 starter
- 50 PartiPrompts plus 50 in-house prompts. CLIP score and VQAScore per prompt, FID over batches, and 3-rater pairwise preference against the prior model.
- A safety classifier pass on every output. Track classifier-flag rate over time; the rate is a regression signal independent of quality.
- Implicit-feedback dashboard (saves, edits, re-rolls) in production. The re-roll rate is the most useful single online signal; it tracks quality regression without needing explicit labels.
Cross-cutting links