Key facts
- Carli Michelle Heller wrote on Sept. 26 that she would "shoot up" the Lee County Sheriff's Office.
- Heller stated she had acquired a new gun by Sept. 27.
- Anthropic's AI platform flagged the user's entries for safety review.
- A human review team at Anthropic reported the user's statements to law enforcement.
- Heller was arrested and charged with making a written threat of violence.
A Florida woman, Carli Michelle Heller, faces a felony charge after her conversations on Anthropic's AI chatbot, Claude, were flagged for threatening content and reported to law enforcement. Heller, of Bonita Springs, allegedly wrote on Sept. 26 that she intended to "shoot up" the Lee County Sheriff's Office. The following day, her account showed she had obtained a new gun, according to investigators.
Anthropic's platform monitors for key phrases and threatening content. The severity of Heller's statements prompted a human review team to escalate the matter to law enforcement. Deputies subsequently identified and detained Heller at her home without incident. She is charged with making a written threat of violence.
Anthropic's consumer terms of service and privacy policy permit the company to report user inputs, outputs, or actions to law enforcement if it believes disclosure is reasonably necessary to prevent serious harm or at its sole discretion. Conversations flagged for safety review are also used to improve the AI's detection capabilities.
This incident follows a similar report in September where Anthropic reportedly referred another user to police after he allegedly made threats against CEO Dario Amodei. OpenAI also has policies in place to refer threats to human reviewers who may then contact law enforcement.
