The article benchmarks Gemma 3 12B and 27B models on Google Cloud TPU v6e to understand how infrastructure performance differs between structurally distinct LLM workloads. The key finding is that for decode-heavy generation tasks, the 12B model scales far better at high concurrency (8.19x normalized throughput at 128 users) while the 27B model plateaus at 4.12x, indicating a strict performance wall past 64 concurrent users. Conversely, for prefill-heavy classification tasks, both models achieve similar peak scaling (around 6.0x to 6.4x), allowing organizations to deploy larger models without a throughput penalty.

The architecture uses a GKE Autopilot cluster with a single-host TPU v6e node pool (2×2 chip topology), vLLM as the serving framework via the vllm-project/tpu-inference hardware plugin, and HuggingFace-hosted models. Two workloads are benchmarked: classification (high input, low output: ~4K input tokens, ~10 output tokens) and generation (low input, high output: ~500 input, ~1K output). Metrics include throughput (requests/sec) and end-to-end latency at 16, 32, 64, and 128 concurrent users.

Results show generation scaling divergence: at 128 users, 12B achieves 8.19x vs 27B’s 4.12x throughput multiplier (normalized to 12B at 16 users). Classification performance parity: both models reach ~6.04x to 6.37x at 128 users. Latency analysis reveals that for 12B classification, E2E latency doubles from 32 to 64 users; for 27B classification, latency remains flat between 32 and 64 users then doubles at 128 users. The article concludes that raw parameter count is not the sole predictor — the interaction between serving framework, hardware topology, and workload token ratios dictates efficiency. Practical advice includes using the 12B model for high-concurrency generation, leveraging larger models for classification, and scaling based on E2E latency rather than CPU/memory metrics. It also recommends aggressive tuning of vLLM parameters like TPU bucket padding to prevent compute waste.