Overfitting

Why does overfitting matter?

An overfit model can look accurate in testing and still fail once it meets production data, which is a more expensive mistake to catch than an obviously bad model.

In life sciences, a model supporting a regulatory submission needs to generalize to data it wasn't trained on, since regulators evaluate performance on held-out or external validation sets, not training accuracy. An overfit model that looks strong internally can fail that external validation step entirely, setting a submission back by however long it takes to retrain and revalidate.

In financial services, out-of-time validation, training on data from one period and testing on a later one, exists specifically to catch this failure mode: a credit or trading model that looks strong on a historical backtest can still be fit to noise that a real market shift will expose.

In the public sector, a forecasting or eligibility model trained on data from one agency or region can overfit to local quirks and fail to generalize when applied elsewhere, a common reason pilot programs don't scale to a full population without retraining.

What causes overfitting?

• Overparameterization: the model has more capacity than the data can support.
• Training for too long: after early gains, further training increasingly fits noise instead of signal.
• Insufficient regularization: without a penalty on model complexity, the model is free to fit every detail of the training set.
• Too little or unbalanced training data: a small or skewed training set doesn't carry enough signal about the true underlying relationship.

What happens when an overfit model reaches production?

A fraud detection model overfit to a training set with a disproportionate share of fraudulent transactions can start flagging nearly everything as fraud, declining transactions for hundreds of thousands of legitimate customers. Fixing that means reducing the model's variance without pushing it so far toward simplicity that it introduces new bias, a balance known as the bias-variance tradeoff.

Overfitting vs. underfitting

Overfitting and underfitting sit at opposite ends of the same tradeoff. An overfit model has learned too much from the training data, including its noise, and performs worse on new data than it did in training. An underfit model hasn't learned enough of the real pattern, and performs poorly on both training and new data. Comparing training and validation performance is usually the fastest way to tell which one you're looking at: overfitting shows a big gap between the two, underfitting shows both scores are weak.

The bias-variance tradeoff

Bias and variance move in opposite directions: raise one and the other falls. Variance is a model's tendency to fit whatever specific data it's given, including noise. Bias is how strongly a model sticks to a simplifying assumption, like a purely linear relationship, regardless of what the data shows.

Some variance is necessary, since the point of machine learning is to learn from data, but a model with enough capacity to fit anything ends up learning very little that generalizes.

How do you address overfitting?

1. Get more or better training data. Adding examples, or balancing classes with oversampling, undersampling, or techniques like SMOTE, usually reduces overfitting more reliably than any other single change.
2. Reduce the feature set. Drop features that add noise rather than signal through feature selection: correlation analysis, mutual information, or recursive feature elimination all work.
3. Cross-validate before deploying. Splitting training data into k folds and rotating which fold is held out for evaluation makes overfitting visible during model validation instead of after deployment.
4. Regularize or ensemble. Add a penalty term to the loss function (L1/L2 regularization, or dropout in neural networks) to constrain how tightly the model can fit training data, or average predictions across multiple models to reduce any one model's overfitting.

FAQ

How do you know if a model is overfitting?

The clearest signal is a gap between training and validation performance: strong accuracy on training data alongside much weaker accuracy on held-out validation data. If both scores are weak, the problem is more likely underfitting than overfitting.

Can a simple model overfit?

Yes, if the training set is small enough. A simple model needs less data to fit a dataset exactly, including its noise, so overfitting isn't only a risk for large, complex models like deep neural networks.

Is overfitting the same as high variance?

They're closely related but not identical. High variance describes a model's general sensitivity to the specific data it was trained on. Overfitting is what happens when that sensitivity gets bad enough that the model captures noise instead of signal and stops generalizing.