PathMoE: Constraining Expert Paths for Better Mixture-of-Experts

Sparse Mixture-of-Experts (MoE) architectures route each token through a subset of experts independently per layer, leading to a massive space of N^L possible expert paths across L layers. However, tokens in practice cluster into a small fraction of these paths that align with linguistic function, while the vast majority remain unexplored. This statistical inefficiency motivates architectures that constrain the effective path space to amplify natural concentration.

PathMoE addresses this by sharing router parameters across blocks of consecutive layers, thereby constraining the path space and amplifying emergent path structure. Experiments on 0.9B and 16B parameter models show consistent improvements in perplexity and downstream tasks over independent routing, while eliminating the need for auxiliary losses. Analysis confirms that PathMoE produces more concentrated path clusters, better cross-layer consistency, and greater robustness to routing perturbations.

The work establishes expert paths as a useful design axis for MoE architectures, complementary to independent routing mechanisms. Builders should consider path-level constraints as a means to reduce statistical inefficiency and improve model quality without additional training overhead.