This is a most excellent place for technology news and articles.
At least you had backup, right?
Oh, yeah, that's right. You were dumb enough to give AI full access to your production system so likely you're dumb enough to not have backups of anything either.
I take it Claude has full access to all of your git repositories as well so that it could wipe those too?
You got what you deserve
Yeah they did, they had plenty of recovery snapshots. That were able to be deleted at a whim and were deleted by Claude! :D
Honestly. At this point, after it having happened to multiple people, multiple times, this is the only appropriate response.
Anyone who lets AI do this is absolutely inept, lazy, or deserving.
In its default configuration, it stops at EVERY STEP. Do you want to run this command, do you want to update this file, here's the file I want to modify and the patch i'm going to use with adds and deletes in green and red.
If you're using it in unsafe permissions mode, click yeah sure allow Claude to run whatever the fuck it wants in this directory, or just hitting yeah sure go ahead every time, it's your own damn fault.
It's self-driving for the terminal. Don't you dare take your eyes off the road or hands off the wheel.
Given that the infrastructure description included the DataTalks.Club website, this resulted in a full wipe of the setup for both sites, including a database with 2.5 years of records, and database snapshots that Grigorev had counted on as backups. The operator had to contact Amazon Business support, which helped restore the data within about a day.
Non-story. He let Terraform zap his production site without offsite backups. But then support restored it all back.
I'd be more alarmed that a 'destroy' command is reversible.
Why aren’t we adding any safeguard to what commands AI models can use?
Idiot forgot --no-preserve-root, what a dumb machine, heh.
Claude code has them, it's just that this guy apparently doesn't know how to do Terraform either
Remember when Gemini got caught in a loop of self-loathing and nuked itself?
The code is cursed, the test is cursed, and I am a fool.
Such venom, of which only a programmer could spew.
Perhaps the A.I. isn't so different from us.
My CTO keeps telling me I need to try agenic coding, and I keep telling him I won't touch shit until I have an isolated VM to use it in, because I'm not letting some fucking clanker nuke my scripts/documentation/mailbox/whatever for no reason.
Too bad there's never any free time to set that shit up. Oh damn........
We used to say Raid is not a backup. Its a redundancy
Snapshots are not a backup. Its a system restore point.
Only something offsite, off system and only accessible with seperate authentication details, is a backup.
AND something tested to restore successfully, otherwise it's just unknown data that might or might not work.
(i.e. reinforcing your point, no disagreements)
3-2-1 Backup Rule: Three copies of data at two different types of storage media, with 1 copy offsite
You either have a backup or will have a backup next time.
Something that is always online and can be wiped while you're working on it (by yourself or with AI, doesn't matter) shouldn't count as backup.
AI or not, I feel like everybody has had "the incident" at some point. After that, you obsessively keep backups.
For me it was a my entire "Junior Project" in college, which was a music album. My windows install (Vista at that time - I know, vista was awful, but it was the only thing that would utilize all 8gb of my RAM because x64 XP wasn't really a thing) bombed out, and I was like "no biggie, I keep my OS on one drive and all of my projects on the other, I'll just reformat and reinstall Windows"
Well... I had two identical 250gb drives and formatted the wrong one.
Woof.
I bought an unformat tool that was able to recover mostly everything, but I lost all of my folder structure and file names. It was just like 000001.wav, 000002.wav etc. I was able to re-record and rebuild but man... Never made that mistake again. Like I said. I now obsessively backup. Stacks of drives, cloud storage. Drives in divverent locations etc.
He did have a backup. This is why you use cloud storage.
The operator had to contact Amazon Business support, which helped restore the data within about a day.
Whoever did this was incredibly lazy. What you using an agent to run your Terraform commands for you in the first place if it's not part of some automation? You're saving yourself, what, 15 seconds tops? You deserve this kind of thing for being like this.
Good. Anyone foolish enough to write code with a slop machine produces only slop. That garbage should've been deleted anyway.
That's entirely ignoring the fact that this person didn't have any backups elsewhere.
If you can't think, you can't code.
Who let's AI anywhere near production environments? Fully deserved
This keeps happening. I can understand using AI to help code, I don't understand Claude having so much access to a system.
You’ve heard of vibe coding. Allow me to introduce despair coding.
Na this is vide ops. Anyone who thought a coding machine could do ops probably assumes anyone who codes can also do ops. It's going to be making the same mistakes that have happened in DevOps.
have you heard of not giving the keys to your wacky robot wizard instead
Im also confused. Do these people not have some sort of version control and backups? Even if the AI did it, no one has backups? Did the ai also delete the backups and repos? If the building burnt down, would they be in the same situation, it just wouldnt make it to the news?
Pretty funny.
You're absolutely right! I made a fatally flawed decision by removing the production environment. The consequences likely have high impact. I'm sorry. Would you like me to log these mistakes to prevent further missteps or would you like me to write up an outline for the redeployment process?
"and database snapshots that Grigorev had counted on as backups" -- yes, this is exactly how you run "production".
We don't need cautionary tales about how drinking bleach caused intestinal damage.
The people needing the caution got it in spades and went off anyway.
Or maybe the cautionary tale is to take caution dealing with the developers in question, as they are dangerously inept.
Nobody wants to point out that Alexey Grigorev changes to being named Gregory after 2 paragraphs?
Slop journalism at its sloppiest. I wouldn't be surprised to find out that this story was entorely fabricated.
Whether human, AI, or code, you don't give a single entity this much power in production.
Happy to see this, because it's fully deserved. Let real coders do the job!
I don't feel an inkling of sympathy. Play stupid games, win stupid prizes.
So no real developer was harmed.
It seems that every few weeks some developer makes this same mistake and a news is published each time.
I'm an engineer using Terraform and Claude Code as well in a much larger and more expensive setup than his.
You do not let Claude Code run terraform apply, it has zero benefits. All it does is that it runs the command and obscures the output. Most of the time is going to be spent in waiting for the automation anyway, most of the effort that you can spare is before running apply.
Also:
applying delete protections to Terraform and AWS permissions, and moving the Terraform state file to S3 storage instead of his local machine
These both take like 20 seconds, and should be in the getting started manual of Terraform and AWS databases respectively. Setting up remote state is 5 minutes in vanilla Terraform, 30 seconds in something like Terragrunt.
Also, use OpenTofu, stop supporting corporate acquisitions, also takes zero effort and money.
And finally:
most sysadmins will spot the baseline issues with Grigorev's approach, including granting wide-ranging permissions to what's effectively a subordinate of his, as well as not scoping permissions in a production environment to begin with.
No, not subordinate. Tool. Two big differences with it. A subordinate might understand more than you do about the code, a tool will guess and rely on you. And the second one is that you practically can't separate your and your tools' permissions, I mean Claude Code will supposedly ask you if it can use some tool or another and you can whitelist actions it can take, but it will never be completely locked out of destroying your database the way you can lock another user out.
Why would somebody trust AI with access to their production servers, and why would that person also not have remote database backups
According to mousetrap manufacturers, putting your tongue on a mousetrap causes you to become 33% sexier, taller and win the lottery twice a week.
While some experts have argued caution that it may cause painful swelling, bleeding, injury, and distress, and that the benefits are yet to be unproven, affiliated marketers all over the world paint a different, sexier picture.
However, it is not working out for everyone. Gregory here put his tongue in the mousetrap the wrong way and suffered painful swelling, bleeding, injury and distress while not getting taller or sexier.
Gregory considers this a learning experience, and hopes this will serve as a cautionary tale for other people putting their tongue on mousetraps: From now on he will use the newest extra-strength mousetrap and take precautions like Hope Really Hard that it works when putting his tongue in the mousetrap.
