The Open ASR Leaderboard has long faced a blind spot: its aggregate Word Error Rate (WER) hides large disparities across speakers. Research has documented higher error rates for Black speakers, women, and non-standard accents, but the leaderboard’s test sets record only what was said, not who said it. To address this, Voice Arena and Hugging Face launch Monsoon, the first Indic-language evaluation sets on the leaderboard—covering Hindi and Indian English.

Monsoon comprises four speaker-disjoint splits: public and private sets for each language, totaling 4,888 speakers across 30 Indian states/union territories and 428 districts for English alone. Each clip carries 12 demographic, geographic, and device attributes, including district, occupation, education, income, handset model, and years in current city. The data was collected from unscripted, dual-channel spontaneous conversations recorded on contributors’ own handsets under real acoustic conditions—deliberately preserving environmental noise and unstable bandwidth. Quality control included language verification, gender confirmation, and detection of played-back audio.

The Hindi sets address an acute problem: orthographic variation. Hindi lacks a canonical spelling, and code-mixed English-origin words have no settled Devanagari form. A single phrase can have ten valid written variants. Traditional WER penalizes systems for choosing a different spelling than the annotator, inflating errors. Monsoon ships a lattice of accepted spellings per span and computes Orthographically-Informed Word Error Rate (OIWER), aligning hypotheses against the entire accepted set. The authors open-source voi-oiwer for reproducibility. Scoring against a flattened single reference shifts rankings arbitrarily, confirming that the lattice removes orthographic noise.

An illustrative analysis on the public Indian English split shows why this metadata matters. Eight top models land between 4.81 and 4.99 WER on the corpus—effectively tied. But per-region, openai/whisper-large-v3-turbo varies by 0.46 points across India’s five zonal councils, while mistralai/Voxtral-Mini-3B-2507 varies by 1.68 points (4.38 Central vs. 6.06 East). The hardest zone differs per model, implicating model bias rather than audio difficulty. None of this would be visible without per-speaker geography.

Private splits are withheld to prevent benchmark-specific optimization. Indian English sets join the main leaderboard’s default column and contribute to the headline Average WER; Hindi appears in the Multilingual tab, ranked only for models supporting all selected languages. The authors explicitly note that these sets do not fix the general problem of unseen diversity—they make it visible. Monsoon is part of Voice Arena’s broader initiative for Global South languages, with future coverage expected.