Automatic speech recognition (ASR) is a worst-case setting for semi-supervised federated learning (SSFL). In SSFL, a teacher model generates pseudo-labels for clients’ unlabeled data, while a small labeled seed dataset is kept on the server to ground training. ASR is especially fragile because pseudo-label errors are not isolated: they compound along the output sequence and across training rounds, so a model trained on noisy labels can diverge. The result has been a large quality gap between SSFL and fully-supervised federated learning.

This paper from Wonho Bae, Zakaria Aldeneh, Martin Pelikan, Jan “Honza” Silovsky, Tatiana Likhomanenko, and Sheikh Shams Azam isolates the two coupled design axes that control this gap. The teacher axis determines which model generates pseudo-labels. The anchor axis determines how the server uses its labeled data to stabilize training. The central claim is that these axes cannot be tuned independently: aggressive teacher choices only help once the anchor is strong enough to keep training on course.

The teacher comparison covers three options. A broadcast global teacher is a single server model fixed within a round, so all clients share the same pseudo-labels. A per-client online teacher uses each client’s own evolving model, which adapts to that client’s data but also inherits that client’s label noise. On its own, the online teacher diverges; each client drifts in its own direction. Once the anchor stabilizes it, however, the online teacher matches or beats the global teacher, decisively in-domain and competitively under domain shift. A transitioning teacher starts with the global model and switches to the online teacher at round r. As the labeled seed grows stronger, the online teacher‘s advantage narrows, and the transitioning teacher matches or beats both fixed choices.

The anchor axis is just as important. The server must keep training on labeled data between rounds; if it stops, the online teacher drifts. The authors report that this interleaving of server-side labeled updates governs convergence more than the seed model itself. Stabilization is highly sensitive to data augmentation and batch size, because those settings control how much input and gradient noise the server injects during its updates. The amount of stabilization required is domain-dependent, governed by the dispersion of the seed data and its overlap with client data.

The resulting recipe improves over the strongest prior method on 9 of 11 domain pairs, with a 20.8% average improvement in-domain and 10.0% cross-domain, narrowing the gap to fully-supervised federated learning. The practical guidelines are direct: pick the teacher according to seed strength and expected domain shift, keep the server’s labeled-data training running between rounds, and tune augmentation and batch size carefully because they determine whether the anchor can hold the online teacher in place.