What just happened? Another incident has taken place that illustrates the need to be careful what you tell AI. A Florida woman is facing felony charges after she used Claude as a diary and allegedly wrote that she planned to “shoot up” the Sheriff’s office. After a human reviewer examined the statements, they were reported to police.
According to the arrest report, Carli Michelle Heller, of Bonita Springs, Florida, wrote on September 26 that she would attack the Sheriff’s office. She later said that she uses Anthropic’s chatbot like a “diary.”
Claude’s safety systems flagged the entry and it was escalated to a human reviewer. After deciding it was a credible threat, the reviewer reported it to law enforcement.
The company says it may share user information in limited emergencies if it believes disclosure is necessary to prevent death or serious physical injury.
Deputies identified Heller and visited her home. She was detained without incident before an LCSO intelligence detective took over the investigation.
Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
Anthropic isn’t going to be taking any chances when it comes to anything it deems a potential threat. Last month, it was reported that OpenAI and Sam Altman are being sued by British Columbia over claims that the company could have prevented a mass shooting in the Canadian province.
The shooter, eighteen-year-old former pupil Jesse Van Rootselaar, had previously been flagged by OpenAI’s safety team for her conversations about gun violence, but OpenAI never alerted police because the conversations did not meet the threshold for legal referral.
In June, Florida also sued OpenAI and Altman, alleging that ChatGPT had contributed to real-world harms, including the 2025 Florida State University shooting.
The latest incident is another reminder to think before you enter something into a chatbot that could get you into trouble. It’s certainly not a private diary whose contents are for your eyes only.
Reports last month revealed that human contractors reviewing Microsoft Copilot’s image editor can see users’ prompts, uploaded photos and AI-generated edits. Documents show that some of those assignments contain sexual, disturbing or potentially illegal material, though the reviewers are not there to flag the content – only to assess whether the output is accurate.
ZDR policies really are a must. If a service doesn’t have it, don’t use it.
ZDR?
Zero data retention. And even that is a pinky promise unless compliance has been audited by an independent 3rd party.
Honestly, if this sort of stuff is a concern for people, I’d investigate using direct API access, or perhaps subscribing to Lumo, which has a better privacy model from what I understand.
Or run it at home on your own rig. Qwen 3.6-35B Is more than capable for this use case. Hell, Qwen 3.5-9B even.
Zero Data Retention
Presumably zero data retention? It seems to me that what you’d want is for it to never be processed off-device in the first place, but I suppose to each their own.
Sounds like a gray area. She didn’t write it with the intention of it being transmitted to anyone. It wouldn’t be any different than writing it in a private diary. The BC lawsuit though puts them in an obvious bind so the law needs to choose a side, is it a private forum or not?
They need to put one of those “I understand” checkbox disclaimers when you start a session. “I understand that what I enter here could be subject to human review and reporting to law enforcement.”
as someone who designed UI/UX, nobody would read it.
what you described is a “legal obstacle” that only helps the company mitigate a future lawsuit from happening.
easier and cheaper to bury it in the TOS/contract and let the lawyers figure it out.
The company says it may share user information in limited emergencies…
IMO sharing the information isn’t so much the issue as “reviewing” it in the first place. She wasn’t publishing a blog.
AI Hyperscaler: My emergency is that we will run out of funding in a year if we don’t [insert morally bankrupt action here] to raise another 10 billion USD.
While a chatbot has been demonstrated not to be a private diary, it will be interesting to see if the courts agree that she intentionally use her diary to communicate or thought that someone else would read it. Like most users of dear diary she probably thought her secrets were safe and her diary would never tell.
This. The law says, “to send, post, or transmit a written or electronic record.” How is talking to AI satisfying the threshold of sending, posting, or transmitting?
How would this conversation NOT classify as “send[ing] a[n] electronic record”? There’s very literally no interpretation that would escape it… this was an online chat bot, that only gets messages (record) that you send over the internet (electronic).
Because the context is likey to be “… to a third party” of a presumably human kind. Human to clanker (which then gets eavesdropped) ought not to qualify.
Ofc letter of law vs spirit of law, variation in jurisdictions, legal interpretation, YMMV etc etc
Not reading into the law… because well. I’m lazy. My query would be if transmitting needs to be to a receiving party (i. e. Threat directly to the police station, or a public group /person that would receive the threat). If this person was using it as a diary, it may warrant that there was never any intention for the threat to be publicized and thus not acted on? Dunno. We’re both probably dumber for writing and reading my comment
Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone
This means a significant portion of Lemmy (and the rest of the internet) are criminals evading justice
great law top shelf
If Florida weren’t already Trump’s safe space, his death threats to dissidents would add more felonies to his list.
Good thing I ain’t in Florida then ain’t it, and if Florida illegally sends their marshals after me then I get some toys.
If you think Lemmy is bad, check out 4chan/b and /pol. We’ve had a few active shooters forecast there first, but it’s ignored in a sea of statements of intent and calls to violence.
An awful lot of people have suicidal or homicidal thoughts cross their mind. Doubly so in an era in which entire demographics have no plausible hope for their future. I, for one, want to see the lifetime appointments of Federalist Society jurists become a liability. Especially, Justice Kavanaugh.
So this incident raises a question of what specific clause of text convinced a human agent to escalate the matter to escalate to authorities who are very fond of using deadly force. Or if this is just a matter of a corporate CYA response.
How FUCKING hard is it to use like, notepad?
Notepad doesn’t answer and doesn’t agree as eagerly.
I was about to say “I know! That’s the beauty of it!” but didn’t they integrate Copilot into Notepad?
use Vim.
Use EDIT.COM for MS-DOS.
I love that Florida Man is a meme for being completely wild, dangerous, and generally criminal, but Florida Woman writes a diary.
No problems here.
She clearly didn’t intent to act on it and didn’t intend for it to be read by anyone else than her AI boyfriend. So it’s not matching the criteria of the law. Legally, she did nothing wrong.
Carli Michelle Heller has really changed since back in the day when she used to play Buffy the vampire slayer on TV.
Dear Claude, I’m aware of a dangerous pedophile’s location, can you report him to the police?
After a human reviewer examined the statements, they were reported to police.
The guardrails are working as intended. ONLY persecute the public with them. Government skynet, and bond villains will have access to non guardrailed models, or any “flagged for review” markers will be ignored to bomb more elementary schools. A sheriff’s office is government power related. School shootings is just a thoughts and prayers issue, if you’re confused about the line between this and other incidents/attitudes we are being spied on for.
“The company says it may share user information in limited emergencies if it believes disclosure is necessary to prevent death or serious physical injury.”
Also keep in mind: Your chat logs are company assets. Even if today’s decision makers believe maintaining user trust is net profitable, that will not always be true. Some day, maybe when the pullback comes and the wolves are at the gates, or when they sell the company, or when a shareholder action asserts fiduciary duty to maximize shareholder value, your chat logs will be sold and mined in the most lucrative way possible. The way that extracts the most value from you, the product.
when a shareholder action asserts fiduciary duty to maximize shareholder value
People really should stop spreading this fiction that there is some legal requirement to maximize shareholder profits embedded in the concept of fiduciary duty.
There’s no law but there is precedent isn’t there?
No good or regularly cited precedent, no, but sadly yes you can point to some cases. It’s worth noting that that even the courts look at those cases like “ew”, or they are only viewed as valid precedent to lay people, with the courts view them completely differently. Dodge v Ford is a common example.
The communication must be made in a manner in which another person may view it.
So she shouldn’t even get charged.
Yeah, I’d wager that is what the case will hinge on. She wasn’t aware anyone could view it. Otherwise it’s incredibly unlikely that she would have made such a statement to Claude at all.
On the other hand though, I’m sure Claude’s parent company and the prosecution will argue that she didn’t have an expectation of privacy where this statement is concerned (given TOS/EULA, in combination with the number of news articles/reports on the subject).
i think anthopic was right to report it, but the cops likely overstepped. if she had “threatened” the president she would have been assesed by ss but likely not arrested. i wonder if the same cops she was “threatening” were the ones she was reported to and arrested by.
This is just ass coverage for the company. They’re not going to pursue it. They just didn’t want to get sued for not reporting it if she did commit a mass shooting. The prosecution though might.
that’s not how privacy works.
