---
title: "15.1 Interpreting and Explaining"
icon: "file-slides"
format:
  revealjs:
    theme: [default, slides.scss]
    scrollable: true
    revealjs-plugins:
      - "fix-plotly-size"
execute: 
  echo: true
---

## High-Stakes Applications {.smaller}

- healthcare
    - Why does it say I'm at high risk for heart disease?
    - Why did my insurance company deny my claim?
- criminal justice (e.g., [ProPublica COMPAS](https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm)))
    - Why was this defendant denied bail?
    - Why target this neighborhood for increased policing?
- finance
    - Why was my loan application denied?
    - Why was my transaction flagged as fraudulent?
- hiring
    - Why was my job application rejected?
    - Why was I not promoted?

## Outline

- Explaining black-box models
  - Feature importance (permutaton)
  - Global explanations (partial dependence)
  - Local explanations (SHAP, LIME)
- Interpretable models
  - Decision rule lists
  - Additive risk score models


# Interpretability vs Explainability

- **Interpretability**: can we understand *how* the model works?
- **Explainability**: can we explain *why* the model made a particular prediction?


# Explaining Black-Box Models

**Black-box model**: a model that is difficult to interpret (e.g., random forest, gradient boosting model, deep neural network)

## Why explain?

- Debugging
- Feature Engineering
- Collaborating with domain experts
- Helping users understand when to trust a model

![](img/husky-vs-wolf.png)

Image from: ["Why Should I Trust You?": Explaining the Predictions of Any Classifier | Abstract](https://arxiv.org/abs/1602.04938)

## Feature Importance {.smaller}

- We gave the model many features, but which ones did it use?
- Simple approach: remove each feature and see how much the model's performance decreases
  - but we'd have to retrain the model many times
  - if two features are correlated, removing one might not affect the model's performance
- Faster approach: **permutation feature importance**: scramble the values of each feature; how much does the model's performance decrease?
- But doesn't fix the problem of correlated features

Read more:

- [Interpretable Machine Learning book chapter](https://christophm.github.io/interpretable-ml-book/feature-importance.html)
- [sklearn User Guide: Permutation feature importance](https://scikit-learn.org/stable/modules/permutation_importance.html)


## Partial Dependence Plots

- How does the model's prediction change (in general) as we vary a feature?
- Individual conditional expectation (ICE) plot: compute the prediction for each row in the test set as we vary a feature
- Partial dependence plot: plot the *average* test-set prediction when we set a feature to a particular value; repeat for different values

![](https://scikit-learn.org/stable/_images/sphx_glr_plot_partial_dependence_004.png)

![](https://christophm.github.io/interpretable-ml-book/images/pdp-bike-1.jpeg)

Read more:

- [Interpretable Machine Learning book chapter](https://christophm.github.io/interpretable-ml-book/pdp.html)
- [sklearn User Guide: Partial dependence plots](https://scikit-learn.org/stable/modules/partial_dependence.html)


## Local Explanations

- For a particular prediction, which features were most important?
- Might be different from the global feature importance
- **SHAP**: Shapley values from game theory
  - intuition: How much would the prediction change if we removed each feature?

![](https://christophm.github.io/interpretable-ml-book/images/shapley-bike-plot-1.jpeg)

Predicting "Man of the Match" award in a soccer game:

![](https://storage.googleapis.com/kaggle-media/learn/images/JVD2U7k.png)

Read more:

- [Interpretable Machine Learning book chapter](https://christophm.github.io/interpretable-ml-book/shapley.html) (source for the bikeshare example)
- Kaggle course (source for the soccer example): [SHAP Values](https://www.kaggle.com/dansbecker/shap-values)
- [SHAP: A Unified Approach to Interpreting Model Predictions](https://arxiv.org/abs/1705.07874)

## Local Explanations (LIME)

- Local Interpretable Model-Agnostic Explanations
- Train a simple model to approximate the black-box model's predictions

![](https://christophm.github.io/interpretable-ml-book/images/lime-tabular-example-explain-plot-1-1.jpeg)

Read more:

- [Interpretable Machine Learning book chapter](https://christophm.github.io/interpretable-ml-book/lime.html)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier | Abstract](https://arxiv.org/abs/1602.04938)
  - See screenshots [here](https://hackernoon.com/dogs-wolves-data-science-and-why-machines-must-learn-like-humans-do-41c43bc7f982)


# Interpretable Models

Instead of making a black-box model interpretable, we can use an interpretable model from the start.

## Simple Models

- Linear regression, logistic regression (with a small number of features and few interactions)
  - *Lasso* Regularization can help select a small number of features
- Shallow decision trees
- Naive Bayes
- k-Nearest Neighbors

Read more:

- [Interpretable Machine Learning book chapter](https://christophm.github.io/interpretable-ml-book/simple.html)

## Decision Rule Lists

- A list of rules that can be used to make a prediction
- Example: CORELS (Certifiably Optimal RulE ListS) 

![](img/explain-corels-1.png)

Read more:

- [Learning Certifiably Optimal Rule Lists for Categorical Data, Angelino et al.](https://www.jmlr.org/papers/volume18/17-716/17-716.pdf) (source of above figure)
- [Interpretable Machine Learning book chapter](https://christophm.github.io/interpretable-ml-book/rules.html)

## Additive Risk Score Models: FIGS

Fast Interpretable Greedy Tree Sums (FIGS): [website](https://csinva.io/imodels/figs.html), [paper](https://arxiv.org/abs/2202.00858). Train a *collection* of decision trees, where the prediction is the *sum* of the predictions from each tree.

![](img/explain-figs.png)

## Additive Risk Score Models: Risk Scorecards

[ustunb/risk-slim: simple customizable risk scores in python](https://github.com/ustunb/risk-slim)

![](https://raw.githubusercontent.com/ustunb/risk-slim/master/docs/images/risk_score_seizure.png)

[Fast and Interpretable Mortality Risk Scores for Critical Care Patients](https://arxiv.org/abs/2311.13015)

![](img/explain-groupfasterrisk.png)

# Going Further

- [Interpretable Machine Learning](https://christophm.github.io/interpretable-ml-book/)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://arxiv.org/pdf/2103.11251.pdf)
- [Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead | Nature Machine Intelligence](https://www.nature.com/articles/s42256-019-0048-x)
[Dissecting scientific explanation in AI (sXAI): A case for medicine and healthcare - ScienceDirect](https://www.sciencedirect.com/science/article/abs/pii/S0004370221000497)
