Metric-Dependent Annotation Saturation for Learning from Label Distributions
A common assumption in annotation is that collecting more labels is always better, but this paper shows that the number of annotators needed to capture disagreement signal depends on the evaluation metric. The authors fine-tune NLI models on label distributions subsampled from ChaosNLI (100 annotators per item) and find that convergence points vary drastically. For entropy correlation—whether the model identifies which items elicit disagreement—the signal requires roughly N=20–50 annotators to stabilize, while KL divergence (distributional match) saturates by N=10, already achieving 87–95% of the improvement across five model seeds. This asymmetry means a uniform annotation budget is wasteful: a budget optimized for distributional match may leave the disagreement-detection task underpowered.
The technical path involves training models on soft labels derived from varying numbers of annotators and comparing them to label smoothing, which the paper argues cannot replicate per-item signal. Across five smoothing intensities, entropy correlation clusters around r=0.45–0.49, whereas soft labels reach r=0.643 (p<0.001). A per-item analysis traces this gap to smoothing’s inability to distinguish ambiguous items from clear ones. The advantage of soft labels replicates across two architectures (DeBERTa, RoBERTa), a non-NLI-pretrained baseline, and an exploratory cross-domain evaluation on content safety. These experiments ground the metric-dependent saturation in a concrete empirical finding rather than a theoretical claim.
For builders, the practical takeaway is that annotation budgets should be metric-aware rather than set uniformly. If your downstream evaluation cares about ranking items by ambiguity (e.g., for active learning or uncertainty estimation), you need a larger annotator pool per item than if your goal is simply to match the aggregate label distribution. The result also reinforces a growing insight: soft labels from real annotators carry item-specific information that synthetic smoothing cannot replicate, so investing in more annotators may be more valuable than engineering better regularizers when disagreement is informative.
