TL;DR: Optimize for recall, because a missed risky clause is legal exposure with no second catch while a false positive costs a reviewer thirty seconds. But cap it with a precision floor (or alert fatigue destroys the recall you bought) and pick the operating point on the PR curve with the customer.
How to approach it
Don't blurt a metric. Open by establishing the cost of each error type and who consumes the output, because that is the whole question in disguise. Then commit to a position, give the operating-point logic, and name the metrics you'd actually report.
A strong answer
First, definitions in one breath: precision is "of the clauses we flagged, how many were risky"; recall is "of the risky clauses, how many did we flag." They trade off against each other via the decision threshold.
The error costs drive everything, and they map cleanly onto a confusion matrix:
In contract review a false negative, a risky indemnification clause sailing through unflagged, can mean real legal exposure, and nobody downstream gets a second chance to catch it. A false positive costs a reviewer thirty seconds of reading. The error costs are asymmetric by orders of magnitude, so I optimize for recall and let precision float as low as reviewers can tolerate. The system's job is to be a safety net under human review, not a replacement for it.
"Maximize recall" alone is a junior answer, because recall of 100% is trivial (flag everything) and then you've built a highlighter, not a classifier. The real design is choosing an operating point: push recall as high as possible, say 95%+ on a labeled eval set, subject to a precision floor that keeps the tool useful. If lawyers see nine false alarms for every true hit, they'll start ignoring flags, and alert fatigue silently destroys your effective recall. I'd validate that floor with actual reviewers, not assume it.
Alert fatigue deserves one line of algebra, because it upgrades "reviewers get annoyed" into a quantity you can reason about: the system's real recall is the product of the model's recall and the reviewer's catch rate on flags. Suppose your 95%-recall threshold buries reviewers in false positives until they skim, and their catch rate on true flags sinks to 60%: system recall is 0.95 x 0.60 = 57%. A humbler 75%-recall threshold whose cleaner queue keeps reviewers engaged at a 95% catch rate delivers 0.75 x 0.95 = 71%. The "worse" model wins by fourteen points, and nothing on your PR curve shows it, because the curve does not know a human is in the loop. The catch rates are illustrative; the multiplication is not, and it is why the precision floor is a recall-protecting device rather than a concession.
Concretely: build a labeled eval set of a few hundred clauses with lawyer-provided ground truth, plot the precision-recall curve, and pick the threshold with the customer, making the tradeoff their informed business decision rather than my silent technical one. Report recall, precision at the chosen threshold, and review burden (flags per contract). If different clause types carry different risk (liability caps versus governing law), use per-category thresholds.
One senior flourish: the asymmetry flips in other deployments. A classifier that auto-sends responses to customers (action without human review) should favor precision, because wrong actions are expensive and missed opportunities are cheap. The meta-answer interviewers want is that the metric follows from the cost matrix and the human-in-the-loop design, never from habit.
The decision in one table:
| Situation | Optimize for | Why (cost of the wrong error) |
|---|---|---|
| Flags feed human review (safety net) | Recall | A missed risky clause is legal exposure with no second catch; a false positive costs a reviewer 30 seconds |
| Recall already high, reviewers ignoring flags | Precision floor | Alert fatigue from unbounded false positives quietly destroys effective recall |
| Output auto-sent, no human in the loop | Precision | Wrong actions are expensive; missed opportunities are cheap |
What interviewers probe next
- "Why not optimize F1?" F1 weights precision and recall equally, which is exactly wrong when error costs are asymmetric. If you want one number, use F-beta with beta > 1, but the PR curve plus a chosen operating point communicates more.
- "Accuracy is 97%, good?" Meaningless under class imbalance. If 3% of clauses are risky, "flag nothing" scores 97%.
- "How do you get labels?" Lawyer-annotated gold set. Measure inter-annotator agreement, because "risky" is partly judgment.
- "How does this change if it's an LLM with a prompt rather than a trained classifier?" Same metrics, but you tune via prompt and criteria changes and you can ask for confidence or rationale. The eval set matters even more.
Common mistakes
- Answering "recall" instantly without the cost-of-errors argument: right answer, zero credit for reasoning.
- Ignoring alert fatigue; unbounded false positives quietly kill real-world recall.
- Quoting accuracy on an imbalanced problem.
- Choosing the threshold yourself instead of with the customer. FDE interviews specifically reward making tradeoffs legible to stakeholders.
Key takeaways
- Recall-first follows from asymmetric error cost, not from a default; say the cost argument out loud.
- A precision floor is non-negotiable: alert fatigue converts unbounded false positives back into missed risks.
- Pick the operating point on the PR curve jointly with the customer, and report flags-per-contract alongside precision and recall.