What "good" looks like
Recommendation is evaluated in two passes: an offline pass that ranks against a temporal split of your own historical interactions, and an online A/B test that measures uplift on the business metric you ship to. No public benchmark works generically. Your interaction logs are the dataset and your business metric is the truth.
The two targets:
- Offline: NDCG@10 or hit-rate@K, computed against a temporal split of historical interactions.
- Online: statistically significant positive uplift on the ship metric (revenue, retention, watch time, conversion) with no regression on the guardrails (diversity, fairness, exposure).
Offline and online often disagree. The right framing treats offline as a filter and A/B as the ground truth. The "good" target is whatever A/B test shows a significant positive uplift on the ship metric without regressing the guardrails.
Eugene Yan's "Patterns for Building LLM-based Systems & Products" has a section on collecting user feedback to build a data flywheel, and it names recommendation systems as a prime example of learning from user interactions. It is the reference for the feedback-loop half of this category, not a source for recsys metrics[1].
flowchart TD
LOGS["Interaction logs"] --> SPLIT["Temporal split, never random"]
SPLIT --> NEG["Hard negatives by popularity or exposure"]
NEG --> OFF["Offline: NDCG@10, hit-rate@K"]
OFF --> IPS["IPS / doubly-robust correction"]
IPS --> FILTER{"Passes offline filter?"}
FILTER -->|no| REJECT["Reject ranker"]
FILTER -->|yes| AB["A/B test on ship metric"]
AB --> GUARD{"Uplift and guardrails hold?"}
GUARD -->|no| REJECT
GUARD -->|yes| SHIP["Ship"]
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Coverage | Catalog items never surfaced | Proportion of the catalog ever recommended |
| Diversity | Over-concentration on a few items | Gini coefficient over the recommendation distribution |
| Novelty | Stale, repetitive slates | How often new items appear in recommendations |
| Position bias | Clicks that reflect rank, not relevance | Tendency of users to click higher-ranked items regardless of relevance, which biases offline metrics from logs |
| Cold-start | Failure on new users and new items | Separate metrics for new users and new items |
| Long-tail uplift | Head-item concentration | Recommendation rate for non-head items |
Offline eval design
Metrics. Use NDCG@10, hit-rate@K, AUC, and MAP on historical interactions.
Split temporally, never randomly. A random split causes data leakage because user preferences and item popularity drift over time, so future events leak into the training data.
Per-user leave-one-out is not temporal enough. It lets training and test periods overlap, so the model still trains on interactions that happened after the test event. Ji et al. show that under leave-last-one-out all four models they tested recommend items that were not yet available at the test timestamp, and that the relative ranking of models becomes unpredictable depending on how much future data leaks[2]. Gusak et al. (RecSys 2025) compare splitting strategies for sequential recommenders and find that outcomes, and model rankings, vary significantly with the strategy. They present a global temporal split, which evaluates on a distinct future period, as the remedy, and note that choosing target interactions and building a validation set consistent with it is still loosely defined[3]. Pick one global cutoff for test, write down your target and validation rules, and keep them fixed across every candidate ranker.
Correct for exposure with counterfactual estimation. IPS (inverse propensity scoring) or doubly-robust estimators are the right move when feasible, because they correct for the selection bias introduced by the current ranker's exposure.
Sample hard negatives. Hard negatives matter here as much as in retrieval. Sampling negatives uniformly from the catalog overstates model performance. Sample by popularity or by current-ranker exposure to make the eval predict real lift.
Figure: Temporal split vs random split for recommendation system evaluation. A random split scatters test interactions through the timeline so training data contains future events (data leakage); a temporal split holds out everything after a cutoff, which is what makes offline NDCG@10 and hit-rate@K predictive of online A/B lift.
Online eval design
A/B on the business metric, not the ML metric. Test on the metric you ship to: revenue, retention, watch time, conversion.
Guardrail diversity, fairness, and exposure. The new ranker may improve the ship metric while concentrating exposure on a small set of items, which is a long-term quality regression.
Alert on selection bias. If the new ranker shifts the candidate pool meaningfully, a candidate-pool shift can invalidate the A/B comparison.
Shankar's "Data Flywheels for LLM Applications" describes the operating pattern: log every output with its metric scores, check each metric's correlation with the KPIs you ship on (she names click-through), and review the low-scoring cases to drive the next change[4]. For a ranker the KPI check is the A/B result above, not the offline score.
Top benchmarks
- Your interaction logs. The dataset that predicts production. Public recsys benchmarks (MovieLens, Amazon reviews) are useful for methodology learning but do not transfer to your domain.
- Offline temporal split of your own logs with hard negatives. The minimum useful eval artifact.
- A/B test rig with proper randomization and guardrails. The truth.
- Cornac: open-source recsys framework with the standard metrics (NDCG, Recall, AUC, MAP, RMSE, MAE) and data splitters. It is actively maintained: v2.6.0 (July 2026) added global temporal and leave-last-out splitting for next-item evaluation, v3.0.0 (August 2026) made
TimestampSplit accept ratio-based splits, and v3.0.1 followed in September 2026[5].
- Elliot: open-source framework that automates the experiment from preprocessing to reporting, with temporal splitting, accuracy, beyond-accuracy, bias and fairness metrics, and Wilcoxon and paired t-tests. Its last tag is v0.3.1 and master has had only documentation fixes since October 2023, so treat it as a stable reference rather than a moving target[6].
- In-house A/B framework: proper randomization, sample-size calculators, and metric pipelines.
- IPS and doubly-robust estimators: off-policy evaluation in-house or via libraries like
Open Bandit Pipeline. That library's default branch has had no commits since November 2022 and its last release, 0.5.7, is from April 2023, so confirm it installs against your numpy and scikit-learn versions before you build on it[7].
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Random offline split | Data leakage; future events leak into training | Always split temporally |
| Optimizing offline NDCG without online validation | The two often disagree | Treat offline as a filter, A/B as the truth |
| Ignoring exposure bias | Your current ranker decides what users see, which biases the logs | Counterfactual estimators or careful negative sampling |
| No diversity or fairness guardrails on A/B tests | A ranker can lift the ship metric and concentrate exposure on head items, a long-term regression A/B alone will not catch | Add diversity and fairness guardrails |
| Reading too much into a single A/B run | The variance is real, and one run's lift can be noise or novelty | Size the test up front, report intervals, and repeat before shipping; count experiments run and learned from, not features shipped[8] |
Canonical reading
Eugene Yan's "Patterns for Building LLM-based Systems & Products" is the most useful single reading for the feedback-loop half of this category. Its user-feedback section separates explicit signals (thumbs, ratings) from implicit ones (interactions) and names recommenders as a prime example of learning from the latter[1]. Read it with Shankar's "Data Flywheels for LLM Applications" for the operational pattern[4]. For the split and leakage questions, read Ji et al. and Gusak et al.[2][3]
Day-1 starter
- Temporal-split historical eval with NDCG@10 and hit-rate. Hard negatives sampled by popularity or current-ranker exposure. The temporal split is non-negotiable.
- A/B harness with guardrails on revenue (or your ship metric) plus diversity plus fairness. Without guardrails, you ship rankers that lift the headline and regress the catalog.
- Counterfactual evaluation (IPS) for offline candidate ranking. The IPS estimator is what closes the gap between offline scores and online uplift.
Cross-cutting links