Transformers Backend for vLLM Matches Native Inference Speed

The transformers library has become the standard for model implementations, but using it inside vLLM traditionally meant accepting slower inference compared to hand-written, model-specific ports in vLLM. Model authors had to write and maintain separate code paths: one clean implementation for training/evaluation in transformers, and another highly optimized one for production serving. This created a tension between code simplicity and deployment speed, discouraging rapid iteration on new architectures.

The new version of the transformers modeling backend for vLLM closes this gap by using torch.fx for static graph analysis and AST-based source code rewriting. At runtime, it detects patterns like QKVParallelLinear and MergedColumnParallelLinear, then replaces them with vLLM’s fused kernels. It also handles expert parallelism for Mixture-of-Experts models like Qwen3-235B. In benchmarks across dense and MoE Qwen3 models (4B, 32B, 235B), the transformers backend meets or exceeds native vLLM throughput. Enabling it is a single flag, --model-impl transformers, and it composes with tensor, data, and expert parallelism.

The practical takeaway is that a single transformers implementation now suffices for both training and ultra-fast inference in vLLM, without rewriting or sacrificing speed. Model authors can iterate in the clean, self-contained transformers style and still get optimized production performance. For builders deploying custom or cutting-edge architectures, this eliminates a major engineering bottleneck. It also means the community can share one authoritative implementation instead of maintaining separate inference-optimized forks.