Interpretability

Why interpretability matters

Simple machine learning models like linear regression are inherently interpretable, but complex models, particularly deep learning systems with many hidden layers, can operate as effective "black boxes," making it difficult to explain results to regulators and stakeholders. Interpretability matters selectively: it becomes necessary once results carry real consequences, like bias, safety, or trust, or the model hasn't yet been validated in production. Beyond compliance, it also gives practitioners a way to catch bias and debug a model's behavior before those consequences show up.

Interpretability, explainability, and transparency

Interpretability, explainability, and transparency get used interchangeably, but they answer different questions.

Types of interpretability

Intrinsic interpretability Models simple enough to interpret directly from their own structure, like decision trees, linear regression, and rule-based models.

Post-hoc interpretability Methods applied after training to explain a model that isn't interpretable on its own, including SHAP, LIME, and permutation importance.

How interpretability is used in practice

1. Train the model.
2. Evaluate its performance.
3. Apply interpretability techniques like SHAP or LIME.
4. Review which features are driving predictions.
5. Validate the results with domain experts.
6. Deploy with ongoing monitoring.

For example, a healthcare team building a hospital readmission model might use SHAP values to confirm that medically relevant factors, not incidental correlations, are driving its predictions before deployment.

Industry use cases

Financial services

SR 26-2 explicitly recognizes interpretability analysis and benchmarking against alternative methods as accepted tools for validating a model, which means a credit or underwriting model's interpretability work can double as compliance evidence, not just an internal quality check. Day to day, it also gives risk teams a faster way to catch a model drifting toward a proxy for a protected characteristic before that shows up in an audit.

Life sciences

Regulatory submissions require reviewers to understand how a model reached a diagnostic or trial-outcome prediction, not just confirm that it performed well in testing, which pushes interpretability into the same validation package as the model's accuracy metrics. This also gives research teams an early signal when a model is relying on a spurious correlation rather than a genuine clinical effect.

Public sector

Agencies making automated eligibility or benefits decisions typically face appeal requirements, so a decision has to be interpretable enough for a caseworker to explain it to the person it affected, not just to another data scientist. That explanation also becomes the basis for internal quality checks before a decision ever reaches appeal.

FAQ

Is interpretability the same as explainable AI (XAI)?

Only partially. XAI is the broader field studying how to make any AI system's behavior understandable to humans; interpretability is one property a model can have within that field, alongside explainability and transparency.

Can deep learning models be interpreted?

Only with the right tools. Deep learning models are complex enough that post-hoc methods like SHAP or LIME are typically needed to approximate which inputs are driving a given prediction.

Does interpretability apply to generative AI models?

Generative and agentic AI raise a different kind of opacity problem, since there often isn't one traceable prediction tied to specific inputs. SR 26-2 places generative and agentic AI outside its model risk framework for this reason, directing institutions toward parallel controls like prompt governance, output monitoring, and agent observability instead of traditional interpretability analysis.

How does interpretability support responsible AI?

Interpretability is one of the practical building blocks of responsible AI: without visibility into which inputs drive a decision, it's difficult to audit that decision for bias or verify it aligns with an organization's AI governance policies.