VSAS-Bench: A New Benchmark for Real-Time Visual Streaming Assistants
Most vision-language model (VLM) benchmarks evaluate on static videos using single-turn question answering, but real-time visual assistants operate under a continuous stream of frames where response latency and behavior across time matter as much as accuracy. The article identifies a fundamental mismatch: existing evaluations measure offline video understanding, but streaming assistants need separate metrics for proactiveness (when the model chooses to respond) and consistency (whether its answer remains stable over a temporal window). Without dedicated benchmarks, builders cannot meaningfully compare streaming VLMs or understand the tradeoffs in their system designs.
To close this gap, the authors introduce VSAS-Bench, a framework with over 18,000 temporally dense annotations across diverse domains and task types. They define standardized synchronous and asynchronous evaluation protocols, plus isolated metrics for distinct streaming capabilities. Using the benchmark, they conduct large-scale evaluations of recent video and streaming VLMs, analyzing the accuracy–latency tradeoff under design choices like memory buffer length, memory access policy, and input resolution. A notable empirical finding: conventional VLMs adapted to streaming settings without additional training can outperform dedicated streaming VLMs—for instance, Qwen3-VL-4B surpasses Dispider, the best streaming VLM on the benchmark, by 3% under the asynchronous protocol.
For builders working on real-time visual assistants, the key takeaway is that streaming performance cannot be inferred from offline video benchmarks. The paper provides a concrete evaluation methodology and reveals that thoughtful system design—particularly around memory buffer and access policy—can narrow or reverse the gap between adapted generic VLMs and purpose-built streaming models. Practitioners should use VSAS-Bench‘s protocols to measure their own latency-accuracy Pareto frontier, and seriously consider adapting a strong conventional VLM before committing to a specialized streaming architecture.
