Benchmarking Frontier ASR on Code-Switched Speech for Voice Agents
Most voice agents are built and tested on monolingual speech, yet over half the world’s population is bilingual and code-switching — alternating languages mid-sentence — is routine in contact centers and IT helpdesks. Transcribing code-switched audio is the first and most critical step in any voice pipeline, since errors propagate into downstream routing, policy enforcement, and resolution. The authors found no existing benchmark for enterprise code-switched ASR, so they built one covering Spanish-English, French-English, Canadian French-English, and German-English with HR and ITSM scenarios. They evaluated seven ASR systems using WER, Semantic WER (SWER), and Answer Error Rate (AER) to capture both raw accuracy and meaning preservation.
The top three models — ElevenLabs Scribe V2, Gemini 3 Flash, and AssemblyAI Universal 3-Pro — showed small deltas between code-switched and monolingual baselines, meaning they handle bilingual input almost as well as single-language input. In contrast, OpenAI Whisper Large V3 Turbo defaulted to translating code-switched audio into English, producing high WER but narrower semantic gaps. The analysis revealed two distinct failure modes: the number of language switches predicts whether an error occurs, while the Code-Mixing Index (CMI) — the density of mixing — predicts error severity. Counterintuitively, errors concentrate on English segments rather than matrix-language segments, possibly because embedded English often carries technical terms or because models struggle with mid-utterance phonological shifts.
The practical takeaway is that code-switching is no longer a universal stress test — top frontier ASRs now handle it with modest penalties. But performance varies significantly across language pairs and models; Spanish-English results do not generalize to German-English. Enterprises must benchmark their actual customer languages rather than relying on overall leaderboards. The authors release their dataset and evaluation harness (AU-Harness) so teams can test their own use cases. The limitations are notable: all audio is TTS-synthesized, not natural speech, and models were tested only in auto-detect mode without language hints — leaving potential further improvements on the table.
