Model Cards for Model Reporting

A trained model is often used by people who did not build it, for purposes its builders did not foresee. The usual release tells them a single accuracy number on a benchmark. This paper proposes model cards: short documents published with a model that state what it is, what it was built for, how it was evaluated and how its performance changes across the people and conditions it will meet.

The card has nine sections. Model details give the developer, date, version, type, license and a contact. Intended use states primary uses and users and also out-of-scope uses, in the spirit of warning labels. Factors list what could change performance: groups of people, the instruments that capture inputs (cameras, microphones), and the environment. Metrics name the performance measures, the decision thresholds and how uncertainty was estimated. Evaluation and training data describe the datasets and why they were chosen. Quantitative analyses report results for each factor on its own and for intersections of factors. The last two sections cover ethical considerations and caveats.

The paper explains which metrics fit which systems. For a classifier, it recommends all four error rates from the confusion matrix (false positive, false negative, false discovery and false omission), with confidence intervals, noting that equality of different rates across groups corresponds to different fairness definitions. For score-based systems it suggests comparing score distributions across groups.

Two example cards make it concrete. A smiling detector trained on the CelebA face dataset reports all four rates by age, gender and their combination at a 0.5 threshold: the false discovery rate for older men was much higher than for other groups. A card for the Perspective API toxicity model, which states it is not intended for fully automated moderation, compares version 1 and version 5 on synthetic sentences mentioning identity terms: version 1 scored terms such as "lesbian", "gay" and "homosexual" poorly, which later versions fixed. Cards should be updated with every release.

A model card is the handoff document for a deployment. When you deliver a classifier, a fine-tuned model or a prompt-based system to a customer, the card is what their risk, compliance and operations teams read. Stating intended and out-of-scope uses in writing prevents the most common failure: the model reused, with no alert to anyone, for a job it was never tested on.

Disaggregated results find the problems averages hide. In our simulated card, overall false discovery was 0.15 and each group looked acceptable, while one intersection of 500 cases sat at 0.40. Break results down by the slices the customer cares about: region, product line, language, document type, customer segment.

Thresholds are part of the model. The paper asks cards to state the decision threshold and why, and ideally show performance across thresholds. A score of 0.5 means nothing until someone picks the operating point and states its error rates.

Regulators and buyers now expect this. Model documentation in this shape shows up in AI governance frameworks, procurement questionnaires and model registries. A card written as part of delivery saves weeks of questionnaires later.

Version the card with the model. The toxicity example shows models change between versions. A card tied to a model version and updated with each release is the record a customer can audit.