Large language models with search tools struggle to produce complete answers for complex multi-hop questions that also involve name ambiguity, such as ‘Which actor from the film Heat won at least one Academy Award?

‘—a question that requires disambiguating multiple films with the same title and reasoning across a large set of actors.

Existing QA benchmarks rarely evaluate both multi-step reasoning and name ambiguity jointly.

To address this, researchers from UC Santa Barbara and Apple introduce DEEPAMBIGQAGEN, an automatic data generation pipeline that creates QA tasks grounded in text corpora and a linked knowledge graph, generating natural and verifiable questions that embed name ambiguity and multi-step reasoning.

Using this pipeline, they built DEEPAMBIGQA, a dataset of 3,600 multi-hop questions, half of which require explicit name disambiguation.

Experiments with state-of-the-art LLMs (including GPT-5) show that even the best models achieve only 0.13 exact match on ambiguous questions and 0.

21 on non-ambiguous questions, highlighting significant gaps in answer completeness. The results underscore the need for more robust QA systems focused on complete information gathering.