What just happened? Another incident has taken place that illustrates the need to be careful what you tell AI. A Florida woman is facing felony charges after she used Claude as a diary and allegedly wrote that she planned to "shoot up" the Sheriff's office. After a human reviewer examined the statements, they were reported to police.
According to the arrest report, Carli Michelle Heller, of Bonita Springs, Florida, wrote on September 26 that she would attack the Sheriff's office. She later said that she uses Anthropic's chatbot like a "diary."
Claude's safety systems flagged the entry and it was escalated to a human reviewer. After deciding it was a credible threat, the reviewer reported it to law enforcement.
The company says it may share user information in limited emergencies if it believes disclosure is necessary to prevent death or serious physical injury.
Deputies identified Heller and visited her home. She was detained without incident before an LCSO intelligence detective took over the investigation.
Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
Anthropic isn't going to be taking any chances when it comes to anything it deems a potential threat. Last month, it was reported that OpenAI and Sam Altman are being sued by British Columbia over claims that the company could have prevented a mass shooting in the Canadian province.
The shooter, eighteen-year-old former pupil Jesse Van Rootselaar, had previously been flagged by OpenAI's safety team for her conversations about gun violence, but OpenAI never alerted police because the conversations did not meet the threshold for legal referral.
In June, Florida also sued OpenAI and Altman, alleging that ChatGPT had contributed to real-world harms, including the 2025 Florida State University shooting.
The latest incident is another reminder to think before you enter something into a chatbot that could get you into trouble. It's certainly not a private diary whose contents are for your eyes only.
Reports last month revealed that human contractors reviewing Microsoft Copilot's image editor can see users' prompts, uploaded photos and AI-generated edits. Documents show that some of those assignments contain sexual, disturbing or potentially illegal material, though the reviewers are not there to flag the content – only to assess whether the output is accurate.
Not the person you're replying to, but I think there's a difference between claiming she wasn't charged with the right thing vs saying no one should be charged for planning a crime. I don't think the intention of the law was to prevent people from writing things wherein they thought they were just venting privately. I understand that her understanding was incorrect, but I think the intention (venting vs creating fear and chaos) is an important factor. I think the law is to prevent posting bomb or shooting threats online or sending them directly to people.
I think there should be legal difference between saying "I hate X so much I just want to kill him" and "what gun would be best to kill a person of Y height and Z weight and what would be the best way to hide their body." I'm familiar with the chat logs from the FSU situation and it wasn't just venting. I'd probably want to see her full logs before making a legally binding decision, but from the summary it does not sound like a credible threat was made. It may have constituted a referral to law enforcement, but based on what I read law enforcement should have just done a check in and likely not charged. They are likely just salty and want to punish people who aren't waving thin blue line flags.
I see your point, but isn't that determination for the court to make after reviewing the chat logs and consulting with competent professionals?
Not that I have much faith in the florida state courts to do the right thing, but honestly if we're gonna say that a person directly claiming they want to shoot a place up doesn't call for some form of intervention, then we need to stop clutching our pearls every time a mass shooting happens.
Note: I'm not expressing a personal opinion that it would be pearl-clutching to ask those sorts of questions in the aftermath of a mass shooting. I'm just pointing out the parallels to this situation and the logical inconsistency in saying "well this one was caught early, so it shouldn't have been; but those ones that weren't caught in time should have been!"
Like pick one or the other.
But also, fixing society and its many systemic issues, improving the quality and availability of mental health services (and reducing the stigma and increasing access), and ceasing to hand out guns like candy would probably be the most effective ways of preventing mass shootings. But it won't be 100% even if we can achieve all that (and that's a big if), so taking threats seriously should still be a part of that strategy either way...
Generally the prosecutor can decide if it's worth it or not to proceed with a case like this, and not all of these even go that far. Cops have some leeway to make decisions on these issues, even if we don't want them to. Cops letting you off with a warning even though you were speeding isn't illegal. If a party was wronged they can press the issue, but in this case the cops were the wronged party so I'm sure the prosecutor wants to stay on their good side. I could also be super wrong and this may be a slam dunk case, but it seems like a huge waste of taxpayer resources that they would not be pursuing if not for the cops being involved as the supposed targets.
Maybe we have different lived experiences, but I certainly have not reported every threat I've overheard. I think people often say things they don't mean, even in mixed company, and taking everything stated as a real credible threat seems misguided. I understand the concern, and obviously mass shootings are a huge issue, but I have friends with no history of violence who don't own any guns say some truly threatening things about certain people and I think sometimes that's just part of life. I'm not saying it's the prettiest part of it, but sometimes we get frustrated and knowing when to actually be concerned is in my opinion an important part of navigating humanity. I've heard parents say that they sometimes want to hit/abandon their kids, and I don't think any less of them as a parent or person. Sometimes we're just not our best selves and so long as we don't let that get the better of us, I think it's ok to acknowledge that. I'm certainly not reporting them to CPS because they told me they thought about just leaving their crying kid at the park one time. I think the same thing applies with violence towards others.