Systematic failures of vision models on semantically coherent subsets, known as error slices, reveal limitations in robustness.

Existing slice discovery approaches model slices as clusters in representation space or combinations of predefined attributes, which work for image-level classification but are insufficient for instance-level tasks like object detection and segmentation.

For instance-level tasks, failures often arise from contextual, relational, and spatially grounded visual patterns.

To address this, the authors propose GH-ESD (Grounded Hypothesis-Driven Error Slice Discovery), a generate-and-verify framework that reformulates slice discovery as grounded hypothesis generation and statistical verification.

GH-ESD constructs relational failure hypotheses using LLM priors and grounded visual evidence, discovers hypothesis slices at the instance level via Vision Language Models, and verifies them through statistical trend analysis over instance-level errors.

They also introduce GESD (Grounded Error Slice Dataset), a new benchmark for instance-level error slice discovery, providing expert-defined and spatially grounded slices derived from detection and segmentation failures.

Experiments show that GH-ESD consistently outperforms baselines, improving Precision@10 by 0.10 (0.73 vs. 0.

63) on the GESD benchmark for detection tasks, while also supporting segmentation scenarios. The identified slices are interpretable and facilitate actionable model improvements.