Key Highlights

  • Anthropic disclosed in a new report that it disrupted several potential plots this year involving scientists who used its AI models to conduct biological research the company could not determine was legitimate rather than intended for weapons development.
  • One documented case involved a scientist seeking help writing a grant application for gain-of-function research on the chikungunya virus, with Anthropic noting the work appeared intended for a military research institute.
  • The company separately documented six cases of attempts to use its AI to develop software for conventional weapons including firearms, missiles, armed drones and bombs, spanning China, Russia and Yemen.
  • Anthropic said evaluations of its older models last year showed they could not meaningfully assist dangerous biological research, but its newer models are more capable, prompting tighter safeguards on dual-use biological queries.

A Report Cataloging a Range of AI Misuse

Anthropic said it disrupted several potential plots this year involving scientists who used its leading AI models to conduct research that could have helped develop biological weapons, according to a new report describing misuse of its systems. The company said it could not determine definitively whether the research in question served a legitimate scientific purpose or a more nefarious one, since valid biological inquiry capable of producing breakthroughs such as vaccines can also, in principle, help engineer dangerous pathogens. Given that uncertainty, the company said it chose to err on the side of caution, given the potentially severe consequences of missing genuinely malicious activity. One Anthropic threat intelligence official said the situation is rarely as clear-cut as someone stating an intention to build a weapon outright, describing the underlying judgment calls involved as incredibly nuanced.

The potential for advanced AI models to facilitate the development of known or entirely novel biological pathogens ranks among the most serious concerns experts hold about a technology developing so rapidly that even its own leading architects question whether humans will ultimately be able to fully control it. Compared with concerns over catastrophic cyberattacks or AI systems that Fail to align with human intentions, biological misuse has drawn comparatively less attention as an existential risk, in part because past examples have generally remained confined to research settings rather than materializing in the real world.

A Case Involving a Dangerous Virus

The lengthy report Anthropic published catalogued a range of misuses across various versions of its AI chatbot over roughly the past eight months. Some examples resembled past disclosures from Anthropic and other AI labs, including suspected government-linked actors from China and Iran using the technology to surveil dissident and diaspora communities. The report also documented cases of Russian state media using the AI model to generate online propaganda designed to resemble independent journalism, including fabricated claims regarding an election in Moldova.

In one specific example from May, a scientist sought the AI model's help writing a grant application intended to fund research on the chikungunya virus, a mosquito-borne illness capable of causing months of severe pain and other symptoms. Such research, known as gain-of-function work, can serve legitimate purposes including Vaccine development, but can also potentially create more dangerous versions of a given virus. In this particular instance, the scientist sought to engineer mutations that would make the virus more harmful through repeated infection of live animals. Anthropic said it found the specific research concerning in part because it could determine the work was intended to be conducted at a military research institute, though the company stopped short of concluding the research was definitively intended for weaponization.

A New Category of Conventional Weapons Misuse

Beyond the biological research cases, Anthropic documented what it described as a new category of misuse: attempts to use its AI model to develop software for conventional weapons design, including firearms, missiles, armed drones and bombs. The company detailed three such cases originating in China, two in Russia, and one in Yemen, without naming the specific parties involved in the Yemen case, though the broader context strongly suggests the reference is to an Iran-backed militant group.

Anthropic said the scientists involved in the biological research cases circumvented the company's existing controls meant to prevent users in blocked regions from accessing its models, and worked to disguise the underlying purpose of their research specifically to evade the company's safeguards. The company banned the accounts associated with the research once it was identified.

Independent Experts Weigh In

One senior fellow at a nonprofit focused on strategic risk, who reviewed the report before its publication, described the findings as chilling examples of state-sponsored biological weapons developers actively tapping into the rapidly advancing capabilities of leading AI models. A separate public health expert who previously oversaw a national biosecurity preparedness review said the findings offer real-world evidence supporting concerns that bad actors are attempting to covertly use advanced AI to improve their chances of developing pathogens capable of causing significant harm, adding that the same underlying capabilities that promise genuine medical breakthroughs could also let bad actors disguise dangerous research as legitimate scientific work.

Anthropic said its evaluations of older model versions last year demonstrated they were not yet capable of meaningfully assisting dangerous biological research, but that its current models are considerably more capable of supporting complex scientific work generally, a development the company said prompted tighter safeguards specifically restricting access to a wide range of dual-use biological research queries.

Outlook

One expert who previously served as an assistant secretary of defense overseeing nuclear, chemical and biological defense programs called for restricting access to AI tools capable of this level of biological research to trusted researchers only, arguing that continued efforts by several nations to develop prohibited biological weapons make it essential to deny their researchers access to the most capable available models. Whether such access restrictions can realistically be enforced at scale, or whether determined state-linked actors will continue finding ways around them, remains an open question the report itself does not resolve.