"Why Should I Trust You?": Explaining the Predictions of Any Classifier

A model can be accurate on its test set for reasons that will not hold in use. The paper's running example is a text classifier separating posts about Christianity from posts about atheism in the 20 Newsgroups dataset: it scores 94% on held-out posts by relying on email headers and author names, and drops to 57% on a new set of web pages. Looking at accuracy alone, you would pick it over a classifier trained on cleaned data that scores lower on the test split but generalizes better.

LIME (Local Interpretable Model-agnostic Explanations) explains individual predictions so a person can catch this. It treats the model as a black box. Around the input being explained, it creates many perturbed versions in a representation people understand: for text, the same document with random subsets of its words removed; for images, the same image with random "superpixel" regions grayed out. It asks the model for a prediction on each, weights each sample by how close it is to the original, and fits a linear model limited to K features. Those K weights, such as "the word 'Posting' pushed toward atheism", are the explanation, and they are meant to be faithful only near that input.

One explanation is not a view of the whole model, and users have limited time. SP-LIME picks a small set of instances whose explanations together cover the features that matter most across the data, without repeating each other, using a greedy algorithm for a submodular objective.

The evaluation has two parts. With simulated users, LIME recovered over 90% of the features that interpretable models actually used, and helped flag untrustworthy predictions and choose between models whose validation accuracy was the same. With real users, Mechanical Turk workers chose the better religion classifier 89% of the time with six SP-LIME explanations, removed non-generalizing words to improve a classifier, and graduate students who saw explanations for a deliberately flawed husky-versus-wolf image classifier noticed that it relied on snow.

Explanations catch leakage that accuracy hides. The religion classifier's 94% came from headers and names. Customer data has the same traps: ticket IDs that encode the outcome, a region field that stands in for the label, a template phrase that only appears in one class. Running a handful of local explanations on a new model is a cheap way to find them before launch.

They help customers trust or reject a model for the right reasons. In the paper, users shown explanations stopped trusting a model that keyed on snow. In a deployment review, a few explained examples per segment do more for a skeptical domain expert than another accuracy table.

Model-agnostic matters in practice. LIME needs only a prediction function. It works the same on a gradient-boosted model, a neural network or a hosted model behind an API, which is often all an FDE has.

Explanations are approximations, so test them. Our run shows that one explanation of a nonlinear model can change with the random seed, the number of samples and the kernel width. Fix the seed, use enough samples, check stability, and do not present weights as exact causes.

Pick examples deliberately. The paper's submodular pick beat random examples by 12 to 14 points in helping users choose models. When presenting explanations to a customer, select examples that cover the important features and segments, not the first few rows.