Florida woman arrested after AI chatbot logs threat, Anthropic reports to police

Anthropic's platform monitors for key phrases and threatening content. The severity of Heller's statements prompted a human review team to escalate the matter to law enforcement. Deputies subsequently identified and detained Heller at her home without incident. She is charged with making a written threat of violence.

Anthropic's consumer terms of service and privacy policy permit the company to report user inputs, outputs, or actions to law enforcement if it believes disclosure is reasonably necessary to prevent serious harm or at its sole discretion. Conversations flagged for safety review are also used to improve the AI's detection capabilities.

This incident follows a similar report in September where Anthropic reportedly referred another user to police after he allegedly made threats against CEO Dario Amodei. OpenAI also has policies in place to refer threats to human reviewers who may then contact law enforcement.