AI hallucination nearly triggers US attack on Chinese ship

US almost attacked Chinese ship because AI hallucination

The United States came very close to launching a military operation against a Chinese vessel after an intelligence report generated with the help of artificial intelligence contained a serious mistake. According to sources cited by CNN, a US Special Operations Command analyst used a chatbot to examine intelligence about the ship’s cargo and produced a report that claimed the vessel was transporting parts for a Chinese nuclear arms program through the Middle East. The report was described as entirely false, yet it prompted the military to prepare for an interception and boarding action with air support before officials realized the error.

The mistake traced back to the chatbot’s tendency to hallucinate, or invent details when its training data does not provide enough context. The system reportedly blended openly available information with classified signals intelligence held by the government and packaged the mixture into a formal assessment. One source told CNN that the AI-powered blunder almost started a war, highlighting how a single synthetic error can escalate into a major international crisis.

AI hallucination problems in military and DoD AI adoption

This incident is not an isolated case of AI hallucination causing trouble. Since the term “hallucination” was named the Cambridge Dictionary word of the year in 2023, similar problems have appeared in fields ranging from journalism and academia to medicine and law enforcement. Researchers have noted that large language models often fabricate content when faced with gaps in their knowledge, and despite various attempts to curb the behavior with special prompts, many experts believe eliminating hallucinations entirely may be impossible.

The Department of Defense has been moving quickly to embed AI across its operations. In January it unveiled an AI acceleration strategy aimed at making all relevant data accessible across federated IT systems so that AI tools can exploit it for missions in every service and component. Defense Secretary Pete Hegseth emphasized that the usefulness of AI depends on the quality of the data it receives and pledged to ensure the department supplies sufficient information.

Military generative AI use fielding models and cautionary guidance

Efforts to field specific AI models have followed. Last December the department announced it would adopt Google’s Gemini for Government as the foundation for a custom platform called GenAI.mil. Last month it added Grok for Government as an alternative option on that platform. Anthropic also provides a tailored version of Claude for use by US intelligence agencies.

The military’s reliance on generative AI is already substantial. In June a Pentagon representative told Congress that the technology assists in drafting congressionally mandated reports and noted that 1.5 million active duty personnel have used the department’s generative AI tools. This widespread adoption shows how deeply the technology has penetrated defense workflows even as concerns about reliability persist.

Earlier guidance called for caution. In 2023 the State Department issued a declaration on responsible military use of artificial intelligence and autonomy, urging that any deployment weigh risks and benefits, limit unintended bias and accidents, and always keep a human in the loop within a clear chain of command. Since then, however, the landscape has shifted. Fully autonomous attack drones have been seen in the Russian war in Ukraine and have been tested by NATO-backed contractors. In March the Department of Defense blacklisted Anthropic because the company objected to having its models used in autonomous weapons systems. A federal judge later ruled that the blacklisting

Frequently asked questions

What led the United States to consider a military operation against a Chinese vessel?

An intelligence report generated with a chatbot hallucinated that the ship was carrying parts for a Chinese nuclear arms program through the Middle East, prompting preparations for interception and boarding with air support before the error was discovered.

Why did the chatbot produce the false report about the ship's cargo?

The chatbot hallucinated details because its training data lacked sufficient context, blending openly available information with classified signals intelligence and packaging the mixture into a formal assessment that was entirely false.

What steps has the Department of Defense taken to integrate AI into its operations?

In January the DoD unveiled an AI acceleration strategy to make all relevant data accessible across federated IT systems so AI tools can support missions in every service and component, and it has adopted Google’s Gemini for Government as the foundation of GenAI.mil, adding Grok for Government as an alternative.

How many U.S. military personnel have used the Department of Defense’s generative AI tools, and for what purpose?

Approximately 1.5 million active duty personnel have used the department’s generative AI tools, which assist in drafting congressionally mandated reports according to a Pentagon representative

Source: Ars Technica