Existing VLM benchmarks evaluate models in offline settings, but streaming VLMs—which continuously process an instruction and an incoming frame stream—require metrics beyond pure video understanding, such as proactiveness (response timeliness) and consistency (response robustness over time). To fill this gap, the authors introduce VSAS-Bench, a benchmark and evaluation framework for Visual Streaming Assistants. VSAS-Bench provides temporally dense annotations (over 18,000 annotations) across diverse input domains and task types. It defines standardized synchronous and asynchronous evaluation protocols and introduces new metrics that isolate distinct streaming capabilities.

Using this framework, the authors conduct large-scale evaluations of recent video and streaming VLMs, examining accuracy–latency trade-offs under key design factors: memory buffer length, memory access policy, and input resolution. A central empirical finding is that conventional (non-streaming) VLMs can be adapted to streaming settings without any additional training, and that these adapted models often outperform dedicated streaming VLMs. For example, Qwen3-VL-4B surpasses Dispider—the best streaming VLM on the benchmark—by 3% under the asynchronous protocol.

The paper appears in the How Far Are We from AGI? workshop at ICLR 2024, with authors from Apple. The work directly addresses a gap in evaluation methodology for real-time visual assistants, offering a standardized way to measure capabilities that offline benchmarks miss.