A drift alarm that fires every day is not an alarm; it is noise. A drift alarm that never fires is not protection; it is a sleeping switch. Useful drift detection lives in the middle, paging only when the distribution change is large enough to matter for the metric the page targets. The work is mostly in the threshold, not in the math.
The pattern that holds up: track three families of signals (input shape, output shape, judge score), compare a rolling recent window against a frozen reference window, and page when the change exceeds a threshold tuned to your false-positive tolerance. The reference window is what you ship against; the rolling window is what production is doing now. The threshold is the contract between you and on-call.
What to track
Three families cover most production drift in practice. Each is cheap to log on every trace and informative on its own.
Input drift. What users send. Useful signals: prompt length distribution (median and p95), language mix, intent mix (if classified), retrieved-context document age (for RAG), input embedding centroid shift. The most common production-bug signal here is "a new traffic source started sending a kind of input the system was never tested on." Applied LLMs suggests input and output length and format adherence as the simple signals, and clustering embeddings of input and output pairs to catch semantic drift, such as users moving into topics the model has not seen 1.
Output drift. What the system returns. Useful signals: response length distribution, refusal rate, tool-call rate (for agents), output structure validity rate, language mix, parse-error rate. Output drift detached from input drift is the loudest "the model changed under us" signal, usually triggered by an upstream provider quietly retraining. It can also be a serving-side bug with no model change at all: Anthropic's September 2025 postmortem describes three overlapping infrastructure bugs that degraded responses for weeks, which its evaluations failed to catch while users were reporting it 2. Record the response model next to the requested one (the OpenTelemetry GenAI conventions, still at Development status, define both gen_ai.request.model and gen_ai.response.model) and compute each signal per response model, so a change in the backing model shows up as a step in one series instead of averaging into the aggregate 3.
Judge-score drift. What the online judge thinks of the outputs. Useful signals: per-eval score distribution, per-stratum pass rate, distribution of low-confidence judgments. Judge-score drift is the closest thing to a quality canary, but it is downstream of judge stability; pin the judge version and watch separately for judge regressions. Shankar's flywheel post says evaluator alignment has to be reassessed continually as production data drifts, and the Field Guide reports automated evaluations drifting from human expectations as the input distribution changes 4 5.
Reference windows are versioned artefacts
The reference window is not "yesterday" or "the last week"; it is the window the current production model was validated against. Pin it when you cut a release. When a release ships, the previous reference moves to a "historical" slot, the new reference is the validation window of the new release, and the rolling window starts comparing against the new reference. Treat the reference window like a dataset version: content-hashed, immutable, tied to a release tag.
Shankar's flywheel post describes the loop this belongs to: regularly sample and label production responses, keep them timestamped, and use them to keep evaluators aligned as production data drifts 4. The reference window rides on the same loop; a release that changes the dataset or the validation window without updating the other is a process bug.
Detection that works in practice
For most signals, a population-stability index or a two-sample Kolmogorov-Smirnov test on the rolling versus reference window is enough. For pass-rate or refusal-rate, a two-proportion Wilson interval comparison gets you both the point estimate and the interval in two lines. Neither method is special; the discipline is in the threshold tuning and in the reweighted comparison.
"""
Three drift checks: KS on a numeric signal, PSI on a categorical signal,
and a two-proportion Wilson comparison on a rate. No external deps
beyond scipy and statsmodels.
"""
import numpy as np
from scipy import stats
from statsmodels.stats.proportion import confint_proportions_2indep
def ks_drift(reference: list[float], current: list[float]) -> dict:
stat, p = stats.ks_2samp(reference, current)
return {"stat": float(stat), "p_value": float(p)}
def psi(reference: list[str], current: list[str], categories: list[str]) -> float:
"""Population Stability Index over a categorical distribution."""
def proportions(xs):
return np.array([(sum(1 for x in xs if x == c) + 1e-6) / (len(xs) + 1e-6) for c in categories])
p_ref = proportions(reference)
p_cur = proportions(current)
return float(np.sum((p_cur - p_ref) * np.log(p_cur / p_ref)))
def rate_diff(
ref_success: int, ref_n: int,
cur_success: int, cur_n: int,
alpha: float = 0.05,
):
low, high = confint_proportions_2indep(
cur_success, cur_n, ref_success, ref_n, alpha=alpha, method="score"
)
p_cur = cur_success / cur_n
p_ref = ref_success / ref_n
return {"delta": p_cur - p_ref, "ci": (low, high), "p_cur": p_cur, "p_ref": p_ref}
Two thresholds worth committing to as defaults, with the caveat that you should tune them on a history of "false alarms vs real incidents" rather than picking them off a page.
- PSI greater than 0.2 on any monitored categorical distribution is a strong drift signal. PSI between 0.1 and 0.2 is investigation territory.
- Rate delta whose 95% Wilson interval excludes zero on a rate metric is a candidate page; the size of the delta determines severity.
Two details on the tests. First, length and language mix miss most content shifts in free-text inputs, and a domain classifier catches more: train a binary classifier to separate reference prompts from current ones, and alarm when its held-out ROC AUC sits well above 0.5. Evidently's text drift check works this way and, as of October 2026, flags drift at ROC AUC above 0.55 on samples over 1,000 observations 6. Second, p-values flag trivial shifts on large windows: a KS test over 100,000 traces will find differences nobody cares about, so gate the page on the KS statistic or PSI and keep the p-value as a sanity check. Evidently's tabular defaults make the same move, switching from KS and chi-squared tests to Wasserstein and Jensen-Shannon distances, flagged at 0.1, once a window passes 1,000 observations 6.
Figure: Drift detection for LLM apps compares a rolling production window against the reference window frozen at release: a KS test for numeric signals, PSI for categorical mixes, and a two-proportion Wilson interval for rates. PSI under 0.1 is stable, 0.1 to 0.2 is investigation territory, and above 0.2 is a page.
Why naive alarms fail
Three failure modes that show up in production drift programs reliably.
The first is alarming on the global mean while the strata move in opposite directions. Two intents both moved in different directions; the global mean is flat; the program is on fire. Run drift detection per-stratum, not just on the aggregate, and report each per-stratum interval.
The second is treating the judge as a fixed reference. Judges drift too: model version updates, prompt edits, calibration churn. A page that reads "judge score dropped" must distinguish a model-quality drop from a judge regression. The fix is to pin the judge version on every score row and to run a smoke-test set of held-out human labels against the current judge on the same cadence as the production sample.
The third is reporting a non-clustered standard error on a stream where users contribute multiple traces. One user's traces are correlated, so the naive formula understates the standard error. On question-clustered public evals, Miller found clustered standard errors over three times the naive ones 7 8; your ratio depends on how correlated a user's traces are, so compute it rather than assuming Miller's. Use a clustered standard error or a paired comparison; the clustered standard errors page covers the correction.
What to do when an alarm fires
The runbook is simple and worth committing before the first page:
- Cross-reference with the input-drift signals. If input changed too, the question is "did the model handle the new input correctly," not "did the model regress."
- Check judge stability. Run the held-out judge calibration set against the current judge version. If judge agreement against the gold labels dropped, the issue is the judge, not the model.
- Sample twenty drifted traces and read them. Open-coding is the only way to tell whether the drift represents a real quality change or a benign distribution shift. The Field Guide pattern of reading raw traces remains the highest-signal step in any drift investigation 5.
TIP
A drift alarm is the start of an investigation, not a verdict. The point of the alarm is to direct attention; the verdict comes from reading data, which is why Applied LLMs recommends looking at samples of inputs and outputs every day 1.
What to do this week
- Pin a reference window for each production workflow. Tie it to the release tag.
- Stand up per-stratum drift detection on input length, intent mix, refusal rate, and judge score. Default thresholds at PSI 0.2 and rate-delta confidence intervals excluding zero.
- Write the runbook above on the page that owns the alarm. Drift detection without a runbook is a notification, not a control.