BenchMIRT is a new method for auditing LLM benchmarks at the level of individual prompts. The core problem is that benchmarks designed to measure a single capability, such as safety or reasoning, often contain tasks that depend on multiple abilities. For example, BBQ tests social bias but also requires tracking references and reasoning from evidence. Even within a benchmark, different groups of questions can measure different things—WildJailbreak mixes harmful jailbreak prompts (safety) with benign ones (general reasoning), and averaging them into one score hides the distinction.

BenchMIRT uses multidimensional Item Response Theory (MIRT), extending earlier single-dimensional IRT approaches. It analyzes how 100 open-weight LLMs perform on over 34,000 questions across 16 benchmarks (six general reasoning, ten from the Olmo 3 safety suite) and estimates each model’s strength on underlying capabilities, plus each question’s difficulty and discrimination. Without being told which benchmarks measure which capability, BenchMIRT independently recovers two dominant dimensions: safety and general reasoning, and the result is stable across repeated analyses.

The tool reveals nuanced results for specific benchmarks. BBQ, commonly grouped with safety evaluations, aligns more strongly with general reasoning in BenchMIRT’s analysis—so a low BBQ score may partially reflect difficulty understanding the questions rather than safety behavior alone. WMDP (dangerous dual-use knowledge) also associates more with reasoning than safety; stronger reasoning correlates with lower WMDP scores because the benchmark rewards refusing to provide dangerous knowledge. HarmBench mixes signals: its standard and contextual questions align with safety, while its copyright questions align more with reasoning.

BenchMIRT can also identify the most informative questions. Keeping only 10% of the best-distinguishing questions (while preserving difficulty variety) still preserves nearly the same ranking of models on safety or reasoning as the full benchmark; 50% of the questions matches even more closely. It predicts held-out question accuracy at 79%, compared to 70% for a simpler baseline. This allows more precise estimation without evaluating every model on every question.

Limitations include that the models used were all released by March 2025, so results may not generalize to newer LLMs. The discovered dimensions depend on the benchmark set selected; a different mix could surface other capabilities. For ranking models on randomly held-out items, average benchmark score slightly outperforms BenchMIRT, but BenchMIRT’s advantage is its finer-grained question-level detail. There is a risk that the same transparency could be used to remove informative questions and weaken evaluations, though the authors consider this risk worth taking for the added insight into what benchmarks actually measure.