Modern Large Language Models achieve strong reasoning capabilities with long Chain of Thoughts, but incur substantial computational cost.

Speculative decoding accelerates inference by using a fast draft model to autoregressively propose tokens that a target model verifies in parallel.

However, token-level speculative decoding struggles in reasoning tasks due to unnecessary rejections caused by token mismatches in semantically equivalent steps.

Step-level semantic verification methods accept or reject entire steps, but existing approaches regenerate many rejected steps, wasting target compute.

The authors propose ARBITRAGE, a step-level speculative generation framework that dynamically routes generation based on the relative advantage between draft and target models.

Instead of a fixed acceptance threshold, ARBITRAGE uses a lightweight router trained to predict when the target model is likely to produce a meaningfully better step.

This routing approximates an ideal ARBITRAGE ORACLE that always chooses the higher-quality step, achieving near-optimal efficiency–accuracy trade-offs.

Across multiple mathematical reasoning benchmarks, ARBITRAGE consistently surpasses prior step-level SD baselines, reducing inference latency by up to ~2× at matched accuracy.

The work was accepted at the ENLSP Workshop at NeurIPS 2024.