Thirty percent disagreement is not a fact about the annotators. It is a fact about the task definition, and on a skewed label set it can mean the labels are worse than chance. The week is spent on the guideline and the gold set, not on the model, because no model trains past the noise in its labels.
Updated Sep 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
Thirty percent disagreement is not a fact about the annotators. It is a fact about the task definition, and on a skewed label set it can mean the labels are worse than chance. The week is spent on the guideline and the gold set, not on the model, because no model trains past the noise in its labels.
The candidate who computes kappa rather than quoting raw agreement, reads the disagreements before proposing a fix, and knows that a model cannot exceed the agreement ceiling of its labels is the one who has done this. The weak answer adds a third annotator and majority-votes, which is a way of averaging three copies of the same ambiguity.
No comments yet — be the first to share your approach.
