A team of researchers (Hala Almaghout, Christian Federmann, and Qin Gao) examines whether large language models can replace human annotators for machine translation (MT) quality evaluation. While LLMs are often proposed as a more efficient and cheaper alternative to human judgment, the authors argue that MT evaluation spans many language pairs, domains, and annotation granularities, so LLM performance must be checked across all of those dimensions before being trusted. The paper focuses on two evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA), and compares LLM outputs against human annotations rather than just measuring self-consistency.

The study uses a long-context test set covering 70 language pairs, plus the publicly available WMT23 and WMT25 datasets. The comparison includes both score-level agreement and error-span-level agreement. The central result is mixed: for some evaluation tasks, the agreement between an LLM and a human annotator is higher than the agreement between two human annotators. However, both human-human and LLM-human agreement vary substantially depending on the annotation scheme, the language pair, and the domain. Because of this variance, the authors conclude that both human and model annotations remain unreliable for most settings.

The paper also identifies distinct failure modes for each side. Human annotators are particularly challenged by fine-grained MQM annotations and by low-resource language pairs. LLMs, in contrast, struggle with minor errors, wrong language variants, and error span annotation. These complementary weaknesses suggest that fully automated LLM evaluation is not yet robust, and that purely human annotation is also far from perfect. The authors point toward human-LLM collaborative annotation pipelines as a way to address the reliability issues, though the paper stops short of implementing or testing such a pipeline.

The overall takeaway is that while LLMs can sometimes outperform human agreement baselines in MT quality annotation, neither humans nor models are consistently reliable across the full space of languages, domains, and annotation granularities. This makes the case for careful validation whenever LLM-based evaluation is deployed, and for designing workflows that combine the strengths of both annotator types rather than treating either as a universal ground truth.