Deep Learning Explainable AI
Deep networks reach impressive accuracy, but their internal workings look like a tangle of millions of numbers. Explainable AI, often shortened to XAI, offers methods that reveal why a model produced a specific answer. Clear explanations build trust, expose mistakes, and help people use predictions responsibly.
The Black Box Problem
A black box takes input and returns output without showing its reasoning. Imagine a bank system that rejects a loan application and offers no reason. The applicant cannot fix anything, and the bank cannot check whether the decision was fair.
Input: income, age, savings, debts
|
[ BLACK BOX ]
|
Output: "Rejected" (but why?)
Why Explanations Matter
- Users trust decisions they can understand
- Developers catch wrong shortcuts the model learned
- Regulators in many regions expect transparency for important decisions
- Experts, such as doctors, need supporting evidence before acting on a prediction
A Famous Example of a Wrong Shortcut
A model learned to tell wolves from huskies. Explanation tools showed it focused on snow in the background and ignored the animal. Every wolf photo in the training set had snow. Without an explanation, nobody would have noticed the flaw.
Global and Local Explanations
Global explanation: "Across all customers, income and debt matter most." Local explanation: "For this customer, high debt lowered the score most."
Global methods describe overall behaviour. Local methods explain a single prediction.
Popular Explanation Methods
Feature Importance
This method shuffles one input feature and measures how much accuracy drops. A big drop means the feature matters.
Shuffle income -> accuracy falls 12% (important) Shuffle zip code -> accuracy falls 1% (minor)
Saliency Maps and Grad-CAM
These techniques highlight the pixels or regions that influenced an image prediction. Grad-CAM paints a heat map over the picture, where bright areas show what the network examined.
Image of a dog -> prediction "dog" Heat map: bright on the face and ears, dark on the grass
LIME
LIME builds a small, simple model around one prediction. It tweaks the input in many ways, watches how the output changes, and fits an easy-to-read rule that approximates the network nearby.
SHAP
SHAP assigns every feature a contribution score for one prediction, based on a fair-credit idea from game theory. Positive scores push the prediction up, and negative scores push it down.
Base score for a typical applicant: 600 + Stable income +45 + Long credit history +30 - High debt -60 Final score: 615
Choosing a Method
| Data Type | Helpful Methods |
|---|---|
| Tables of numbers | Feature importance, SHAP, LIME |
| Images | Grad-CAM, saliency maps |
| Text | LIME, attention visualization, SHAP |
Limits of Explanations
- An explanation approximates the model and is not a perfect window
- Different methods can give different answers for the same prediction
- A convincing story does not prove the model is correct
- Simple models such as decision trees stay easy to read from the start
Good Practices
- Choose an interpretable model when accuracy stays similar
- Test explanations on known cases to confirm they make sense
- Show explanations in plain language to the people affected
- Keep humans in charge of high-stakes decisions
Key Terms
- Explainability - the ability to describe why a model gave an output
- Black box - a model whose reasoning stays hidden
- Grad-CAM - a heat map of important image regions
- SHAP - feature contribution scores for one prediction
- LIME - a local, simple approximation of a complex model