Model Drift

Types of model drift?

- Data drift occurs when the distribution of input features changes over time, a shift in customer demographics, a seasonal pattern, or a new data source feeding the pipeline.

- Concept drift occurs when the relationship between inputs and the target itself changes, the definition of what counts as spam, fraud, or a qualified lead shifts even though the input data looks the same.

- Prediction drift refers to a shift in the distribution of the model's outputs rather than its inputs.

- Performance drift refers to the measured drop in accuracy or another evaluation metric itself.

Prediction drift and performance drift are usually the earliest observable symptoms; data drift and concept drift are the underlying causes a team investigates once drift is detected.

Why model drift matters

A model trained on data from several years ago carries assumptions that stop holding as the world changes. Spam filters are the clearest example: models learn to flag phrases like "lottery" or "urgent wire transfer," but spammers adapt faster than most retraining cycles, and detection quality erodes quietly until someone notices the miss rate climbing.

Left undetected, drift doesn't fail loudly. It fails as a slow accuracy decay that looks like normal variance until the cost shows up in a bad credit decision, a missed fraud case, or a misrouted support ticket.

How do you detect model drift?

• Continuous evaluation. Validate model performance on a rolling basis against newly labeled data, comparing recent results to the model's original baseline. See model evaluation for the metrics this typically relies on.
• Population Stability Index (PSI). Quantifies how far a feature's distribution has moved between two time periods. Below 0.1 signals negligible change, 0.1 to 0.2 signals moderate change, and above 0.2 signals a distribution shift worth investigating.
• Z-score comparison. Compares the distribution of live data against training data; a z-score beyond ±3 indicates the underlying data has likely changed.

Detecting drift is one step in a larger cycle: monitor for early signals, detect a confirmed shift, investigate the cause, retrain the model, and redeploy it. This is also where drift detection connects to MLOps practices more broadly, since retraining and redeployment are operational processes, not one-time fixes.

Domino's model monitoring capability runs these checks automatically against deployed models and alerts teams when drift, divergence, or data quality thresholds are exceeded, so retraining decisions happen on evidence instead of a support ticket backlog.

Industry use cases

Financial services. SR 26-2 ties monitoring frequency to model materiality and change velocity rather than a fixed annual cycle, which means a credit model with fast-moving inputs needs drift detection running continuously, not reviewed once a year.

Life sciences. Models supporting clinical or regulatory decisions typically require revalidation whenever the underlying population or protocol shifts, so drift detection functions as the trigger for a formal revalidation review rather than a standalone monitoring exercise.

FAQ

What causes model drift?

Model drift is caused by the real world changing after a model is trained, whether that's a shift in the input data (data drift) or a shift in what the predicted outcome actually means (concept drift). It isn't caused by a flaw in the model itself.

What's the difference between model drift and data drift?

Data drift is a change in the input feature distributions. Model drift is the resulting decline in predictive performance, which data drift can cause on its own or alongside concept drift.

How often should teams check for model drift?

There's no universal cadence. Teams with high-velocity data (fraud, pricing, real-time recommendations) typically monitor continuously; teams with slower-moving inputs can check on a fixed schedule and add continuous monitoring only where materiality justifies it.

Can model drift be prevented?

Not entirely. Drift is a function of the real world changing, not a flaw in the model. The practical goal is early detection and a fast retraining or rollback path, not prevention.