Evaluating a Model in Production: What Offline Metrics Miss

TL;DR

An offline metric measures a model on a frozen sample of past data. It says little about whether the model works once real traffic, delayed labels, feedback loops and changing inputs arrive. This note lists the specific gaps and the checks that address each one: input and prediction monitoring, delayed-label evaluation, shadow deployment, controlled experiments and per-slice reporting. It contains no benchmark numbers by design; the point is the method.

Every team that ships a model has an offline evaluation, and most offline evaluations are honest. The held-out set is clean, the metric is computed correctly, the number goes in the launch document. Then the model goes live and someone asks a simple question: is it working? The offline number does not answer it. It was never meant to.

This note is about the distance between those two questions. We are not arguing against offline evaluation, which remains the cheapest and fastest signal a team has. We are arguing that it is one instrument among several, and that most production surprises come from things it cannot see.

What an offline metric actually measures

An offline metric is computed on a fixed dataset, collected at some point in the past, under some sampling procedure, with labels assigned by some process. Each of those clauses is a place where production can differ.

Collected in the past. The world moves. The test set describes the world as it was when the data was gathered.
Under some sampling procedure. The test set is a sample of what the team could log, which is not always a sample of what users will send.
With labels from some process. Offline labels often come from annotators or from a curated pipeline. Production ground truth, when it exists at all, often comes from somewhere else, with different noise and different definitions.

A model that scores well offline has shown it can reproduce those labels on that sample. Whether that transfers is an empirical question about production, and it has to be answered with production data. For how to read the offline number itself, including sample size and variance, see our earlier piece on why accuracy benchmarks mislead.

The gaps, one by one

Gap	What happens	What surfaces it
Covariate shift	The inputs users send drift away from the training and test distribution.	Input feature monitoring against a reference window.
Label shift	The mix of true outcomes changes, so a fixed decision threshold behaves differently.	Prediction-distribution monitoring; delayed-label evaluation.
Concept drift	The relationship between inputs and the correct answer changes.	Delayed-label evaluation on recent data.
Feedback loops	The model's outputs alter the data it will later be trained or judged on.	Holdouts and randomised exposure.
Pipeline mismatch	Features are computed differently online and offline (skew).	Logging served features and replaying them offline.
Aggregate hides slices	A good average conceals a subgroup where the model fails.	Per-slice reporting with minimum sample sizes.
Metric is a proxy	The offline metric is not the outcome the product cares about.	Controlled experiments on the real outcome.

Figure 1. A taxonomy of the gaps between an offline evaluation and production behaviour, with the check that tends to expose each one. The categories overlap in practice.

The label-shift entry has a formal treatment. Lipton, Wang and Smola's ICML 2018 paper, "Detecting and Correcting for Label Shift with Black Box Predictors", studies the case where the distribution of labels changes while the class-conditional input distribution stays fixed, and proposes an estimator that works from a black-box classifier's predictions. It is a useful reference for why prediction-distribution monitoring can carry information even when you have no fresh labels.

Delayed and missing labels

The hardest practical problem is that production often has no immediate ground truth. A fraud model learns weeks later whether a transaction was disputed. A recommendation model never learns what the user would have done otherwise. A demand forecast is judged only when the period it covered has ended.

This forces a two-tier structure. In the short term, monitor what you can see without labels: input distributions, output distributions, confidence scores, latency, error rates and null rates. These do not tell you the model is right, but a sudden change in any of them tells you something upstream moved. In the longer term, join predictions to outcomes as labels arrive and compute the real metric on a lagged window. Treat the lag as a property of the system and write it down. A dashboard that mixes last hour's traffic with last month's labels is easy to misread.

Selective labels make this harder. If only the cases the model approved ever generate an outcome, then the labelled data is not a random sample of what the model saw, and metrics computed on it are biased toward the cases the model already handles. Randomised exploration on a small fraction of traffic, where the model's decision is occasionally overridden, is the standard way to obtain unbiased outcomes, at a cost the team has to accept knowingly.

Shadow, canary and controlled experiments

Three deployment patterns answer different questions, and confusing them is a common source of false confidence.

Shadow mode. The new model receives live requests and logs its outputs, but users never see them. Shadow mode tests the serving path and the input distribution against real traffic and lets you compare the new model's outputs with the current one's. It cannot tell you how users react, because they never see the output.
Canary release. A small share of traffic goes to the new model. This catches crashes, latency regressions and gross behavioural errors early, with a bounded blast radius. A canary is a safety mechanism, not a statistical test, unless sized and analysed as one.
Controlled experiment (A/B test). Traffic is randomised between models and the outcome the product cares about is compared. This is the only one of the three that measures causal impact on that outcome. Kohavi, Tang and Xu's Trustworthy Online Controlled Experiments (Cambridge University Press, 2020) is the standard practical reference on running them without fooling yourself.

The key move is matching the pattern to the claim. "The new model does not break serving" is a shadow or canary claim. "The new model improves the outcome" is an experiment claim, and no amount of offline improvement substitutes for it when the offline metric is only a proxy.

Slices: where averages lie

A single aggregate number weights every example by how often it occurs. A model can improve on the common case and get worse on a rare but important group, and the average will barely move. Production monitoring should therefore report the metric per slice: by region, device, input length, language, customer segment, time of day, or any partition the domain suggests.

Slicing has its own statistical trap. Small slices produce noisy estimates, and looking at many slices guarantees that a few will look bad by chance. The report needs a minimum sample size per slice and a habit of asking whether a flagged slice is consistently worse over several windows before acting. The sketch below shows the shape of such a report. It is illustrative, not a library.

import pandas as pd

def slice_report(df, slice_cols, label_col="label", pred_col="pred", min_n=200):
    """Per-slice error rate for the rows that already have labels.

    df must contain only labelled rows. Slices below min_n are reported
    as 'insufficient data' rather than as a number, to avoid over-reading noise.
    """
    rows = []
    for col in slice_cols:
        for value, g in df.groupby(col):
            n = len(g)
            err = (g[label_col] != g[pred_col]).mean() if n >= min_n else None
            rows.append({"slice": f"{col}={value}", "n": n, "error": err})
    return pd.DataFrame(rows).sort_values("n", ascending=False)

Proxies and Goodhart's problem

Most offline metrics are proxies for the outcome the business cares about. Click-through rate stands in for satisfaction, F1 for downstream decision quality, mean error for whether the forecast is useful for planning. Proxies are convenient because they are computable, and dangerous because optimising a proxy hard enough separates it from the target. When a team iterates for months against a fixed offline metric, some of the gains are real and some are fit to the quirks of that metric and that test set.

Two practical defences. First, keep more than one metric and watch for divergence: if the primary metric rises while a secondary, differently constructed metric falls, investigate. Second, refresh the evaluation set regularly with recent production samples, so the test set does not freeze at the moment the team started tuning against it.

A production evaluation checklist

Google's "The ML Test Score" (Breck, Cai, Nielsen, Salib and Sculley, IEEE Big Data 2017) presents a rubric of 28 tests and monitoring needs for production readiness, and is worth reading in full. The broader failure modes of ML systems, including feedback loops and undeclared consumers, are laid out in Sculley et al.'s "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015). Our own condensed checklist, drawing on those ideas, is as follows.

Log the exact features served with each prediction, plus a model version and a request identifier.
Define, in writing, what ground truth is, how long it takes to arrive, and how predictions get joined to it.
Monitor input distributions, output distributions and null or default-value rates against a stated reference window, with alert thresholds agreed before launch.
Compute the real metric on a lagged window as labels arrive, and display the lag next to the number.
Report per-slice metrics with minimum sample sizes and repeat-window confirmation.
Use shadow mode for serving-path checks, canaries for blast-radius control, and randomised experiments for claims about outcomes.
Keep a small randomised holdout or exploration slice so that you can measure the model against a baseline rather than only against itself.
Refresh the evaluation set from recent production data on a schedule, and record which version produced each reported number.
Decide the rollback trigger in advance.

When offline evaluation is enough

Not every model needs all of this. A batch model that runs once, whose output a human reviews, in a domain where the data distribution is stable, may be adequately covered by a solid offline evaluation and a manual spot check. The investment in production evaluation should scale with the cost of a silent failure, how fast the data moves, and how long the feedback lag is. A model whose mistakes are cheap and immediately visible can lean on complaints. A model whose mistakes are expensive and discovered late needs the instruments above.

Limits of this note

This is a conceptual overview, not a benchmark and not a survey of vendor tools. We have not attempted to quantify how often each gap causes incidents, and we do not offer thresholds, because the right values depend on the system. The cited sources describe the ideas; how they apply to a given stack is a judgement each team has to make and revisit.