U.S. officials aborted an armed operation against a Chinese vessel this spring after discovering that the intelligence driving it had been hallucinated by an AI chatbot, CNN reported Friday. The intelligence report, circulated during the war with Iran, claimed the vessel carried components for a nuclear weapons program. A Special Operations Command analyst used the AI chatbot to synthesize open source data with classified signals intelligence, and the chatbot misidentified the ship’s cargo manifest. The analyst then used the tool a second time to format the erroneous findings into an official-looking summary that moved across command channels.

The near-miss comes as the U.S. military races to integrate AI to shorten its kill chain, but it illustrates how AI errors can travel up the chain of command before being questioned. Jake Steckler, a GovAI research scholar and U.S. Army veteran, said service members must understand the uncertainty inherent to LLMs, especially for decisions involving targeting, intelligence analysis, or operational planning that could lead to use of force. He called for more safeguards rather than avoiding AI, arguing that prioritizing adoption speed above all else will produce incidents that erode trust and ultimately slow adoption.