Over half the world’s population is bilingual, and code-switching—switching languages mid-sentence—is common in casual conversation, contact centers, and IT helpdesks. Yet few benchmarks evaluate how voice agents handle code-switched speech in enterprise settings. This article benchmarks seven ASR systems on a synthetic dataset of code-switched utterances covering four language pairs (Spanish-English, French-English, Canadian French-English, and German-English) in HR and IT scenarios. The authors release their benchmark and data through AU-Harness.

The dataset was built by starting from parallel English and non-English utterances, filtering for good code-switching candidates (12–40 words, at least three switchable content words, no entity-dominated text), then using GPT-5 with a persona prompt to generate code-switched text, followed by an LLM verbalization pass and ElevenLabs Multilingual V2 TTS. Each utterance was reviewed by a native-speaking linguist. The final dataset contains 259–298 records per language pair.

Three metrics are reported: Word Error Rate (WER), Semantic Word Error Rate (SWER, using Gemma-4-31B as judge), and Answer Error Rate (AER, measuring downstream comprehension via three questions per utterance following Bhushan et al.). The models evaluated include ElevenLabs Scribe V2, AssemblyAI Universal 3-Pro, Deepgram Nova 3, Gemini 3 Flash, Mistral Voxtral Small, Nvidia Parakeet TDT 0.6b, and OpenAI Whisper Large V3 Turbo.

Findings: Scribe V2 and AssemblyAI are top on raw WER, with Scribe taking narrow leads. Gemini 3 Flash follows closely. On semantic metrics (SWER and AER), Scribe V2 remains first, but Gemini 3 Flash outperforms AssemblyAI on AER, likely due to its optimization for language understanding. Whisper ranks last on WER because it defaults to translation rather than transcription on code-switched audio, though its semantic gap narrows. Deepgram Nova-3 sits mid-tier on SWER but ranks last on AER, indicating it misses critical details.

To isolate the cost of code-switching, the authors compared WER deltas between code-switched audio and monolingual baselines (matrix language only and English only). Scribe V2, Gemini 3 Flash, and AssemblyAI show the smallest deltas overall; Scribe V2 even outperforms its own L2 baseline. Lower-ranked models degrade more substantially, confirming that code-switching exposes robustness differences. The switching penalty is larger relative to English than to the matrix language (since the matrix baseline is already harder). Whisper uniquely performs better on code-switched speech than on monolingual L2, again due to translation.

Analysis of error causes used a two-part regression model (logistic for error occurrence, OLS for error magnitude). The number of language switches is most consistently associated with error occurrence (especially for French-English). For error magnitude, the Code-Mixing Index (CMI)—proportion of words from the secondary language—is the stronger predictor (especially for German-English). Per-language WER analysis shows that errors concentrate on the English portions of utterances, not the matrix language. The authors hypothesize this may be due to technical vocabulary or the difficulty of adapting mid-utterance.

Limitations: The benchmark is fully synthetic (TTS-generated), not natural bilingual speech. All models used auto language detection; forced language tokens might improve results. Per-language WER excludes insertions.

Conclusion: Top frontier ASR systems handle code-switched speech with surprisingly small penalties, but performance varies substantially across models and language pairs. Enterprises must benchmark the specific languages their customers use before making production decisions.