Long video summarization is deceptively hard for multimodal large language models (MLLMs). The core tension is that these models must maintain both semantic accuracy and temporal fidelity across extended video durations, yet standard benchmarks rarely test for fine-grained time awareness. The authors of this paper introduce LVSum, a human-annotated benchmark specifically designed to expose this gap. It consists of 72 diverse videos averaging 16 minutes each across 13 domains, with up to 10 human-written summaries per video that include explicit temporal references. This setup directly challenges models to not just describe what happened, but when it happened, a capability that existing evaluations largely ignore.

The paper runs a comprehensive evaluation of both proprietary and open-source MLLMs using new LLM-based metrics for content relevance and modality coherence, alongside standard automatic metrics. Three concrete findings stand out. First, transcripts contribute substantially more to summarization quality than visual frames alone, suggesting current models lean heavily on language signals rather than integrating visual temporal information. Second, there is a significant and persistent performance gap between model-generated summaries and human-written ones, especially in temporal grounding tasks. Third, the models show systematic weaknesses in instruction adherence and cross-modal coherence, meaning they often fail to follow specific temporal instructions or align what they see with what they hear over long sequences.

For builders and researchers, the takeaway is that current MLLMs are not yet trustworthy for applications requiring temporally precise video understanding, such as surveillance review, meeting summarization, or content moderation. The paper makes a strong case that transcript-only approaches are a pragmatic baseline, but real progress will demand architectures or training objectives that explicitly reward temporal reasoning rather than relying on frame-level spatial shortcuts. If you are building a video pipeline, this benchmark offers a concrete way to measure whether your model actually knows when things happen, not just what happens.