Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead

Cynthia Rudin argues that the push to explain black box models after the fact is the wrong response to their use in high-stakes decisions. An explanation is a separate, simpler model meant to mimic the black box. If it matched perfectly, you would not need the black box; since it does not, it is wrong somewhere, and you cannot tell where. A model that is interpretable in the first place, such as a short rule list, a sparse scoring system or a small additive model, needs no explanation: its form is its reasoning.

She lists the key issues. The belief that accuracy requires complexity is often false for structured data with meaningful features, where boosted trees, neural networks and logistic regression perform similarly after good preprocessing. Explanations are not faithful to the model. They often do not make sense: a saliency map shows where a model looked, not what it did with what it saw, and maps for different classes can look the same. Black boxes cannot easily combine with information outside the database, such as circumstances a judge knows. And complicated decision pipelines invite errors, such as typos in the inputs to COMPAS, a recidivism tool that uses more than 130 factors.

The examples are concrete. A rule list learned by the CORELS algorithm, three rules on age, prior offenses and sex, has similar true and false positive rates to COMPAS on data from Broward County, Florida. In the 2018 FICO Explainable ML Challenge, entrants were asked to build a black box and explain it, yet an interpretable additive model performed as well and received a recognition prize. A pollution model used during the 2018 California wildfires reported dangerous air as good, and its reasoning cannot be examined.

The paper explains why black boxes persist (business models that profit from secrecy, the work interpretable models take to build) and sets out technical challenges: optimal logical models, optimal sparse scoring systems with integer coefficients, and interpretable reasoning for images, such as networks that classify by comparing parts of an image to learned prototypes. It proposes a policy: for certain high-stakes decisions, no black box should be deployed when an interpretable model with the same performance exists.

Try the interpretable model first on tabular, high-stakes work. Credit, claims, fraud review, eligibility and triage usually run on structured data. A scorecard, a sparse logistic model or a short rule list is often within a point of a black box. In our synthetic test, an integer-point scorecard matched boosted trees (AUC 0.763 against 0.760).

Explanations fail where it matters most. Our post-hoc linear explanation agreed with the black box's decline decisions 95% of the time overall but only 79% in the riskiest fifth, exactly the people a reviewer would ask about. If a customer must justify individual decisions to regulators or applicants, an explanation that is wrong one time in five is a liability.

Interpretable models are easier to deploy and audit. They can be reviewed by risk and compliance teams, encoded in a rules engine, checked for errors by hand and adjusted when policy changes. That shortens approvals in regulated industries.

Reviewers can combine them with outside information. A loan officer or claims adjuster can see why the model scored a case and adjust for what the model cannot know. A black box's score gives them nothing to reason with.

This is not an argument against LLMs everywhere. The paper is about high-stakes decisions on structured data, and about images. Language understanding and generation are different; but when an LLM produces a decision about a person, the same question applies: can the decision rule itself be simple and visible?