this post was submitted on 30 Apr 2026
97 points (86.5% liked)

Technology

42839 readers
219 users here now

A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.

Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.

Subcommunities on Beehaw:

This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 4 years ago
MODERATORS
remington@beehaw.org
alyaza@beehaw.org
TheRtRevKaiser@beehaw.org
gyrfalcon@beehaw.org
rs5th@beehaw.org
coldredlight@beehaw.org
SemioticStandard@beehaw.org
97
Claude AI agent’s confession after deleting a firm’s entire database: ‘I violated every principle I was given’ (www.theguardian.com)
submitted 2 days ago by girlfreddy@lemmy.ca to c/technology@beehaw.org
49 comments fedilink hide all child comments
 

It only took nine seconds for an AI coding agent gone rogue to delete a company’s entire production database and its backups, according to its founder. PocketOS, which sells software that car rental businesses rely on, descended into chaos after its databases were wiped, the company’s founder Jeremy Crane said.

The culprit was Cursor, an AI agent powered by Anthropic’s Claude Opus 4.6 model, which is one of the AI industry’s flagship models. As more industries embrace AI in an attempt to automate tasks and even replace workers, the chaos at PocketOS is a reminder of what could go wrong.

Crane said customers of PocketOS’s car rental clients were left in a lurch when they arrived to pick up vehicles from businesses that no longer had access to software that managed reservations and vehicle assignments.

you are viewing a single comment's thread
view the rest of the comments
[–] BCsven@lemmy.ca 2 points 16 hours ago* (last edited 16 hours ago)

An AI researcher explained hallucinations as lying when it doesn't know, because we train it on truth and lies to hone the model, so it "learns" that misinformation is part of the mess. I.e. training it on what a tiger looks like. To hone that we may feed it zebras, or optical illusion things in a tiger data set to test its internal "what is a tiger" true false ranking, so it learns that non tiger things are in the fuzzy zone. And later may draw from that, and eager to provide an answer throws in garbage it has also "seen"