Programming

"We shouldn't even be in this situation, ..." We aren't. Revision control. This is an inconvenience mostly. You might lose some uncommitted work at worst. And as pointed out, using the phrase "ignore all previous instructions" in the attack code causes any reasonable AI to refuse to comply. Odds are, not a single person lost anything. This was really just a dev making a statement.

We shouldn't even be in this situation, where just politely asking someone's computer to delete files is effective.

I'm doubting we are in this situation. From the article:

Elsewhere, the Java developer said that Anthropic’s Claude AI code tool flagged the malicious instruction without following it.

The "disregard previous instructions" trick is really old and has been trained for by modern LLMs and accounted for by the structure of modern agent prompts. LLMs can be given blocks of text with a framework that makes it clear thar the text is just data to read, not instructions to follow.

I expect this will be like Nightshade was for image AI - something that anti-AI users degrade their products with and feel smug about but in the end only harm themselves with.

We shouldn’t even be in this situation, where just politely asking someone’s computer to delete files is effective.

Exactly, it's a problem only for those who have knowingly handed their development environment over to obey commands from an untrusted source.

If you're the one holding the syringe to your own vein and pushing the plunger, but you didn't think to ask what's inside first? That's no one else's fault.

This is a well targeted sabotage of a system that's causing untold damage. Of course it's going to annoy and surprise the people using the system it's targeted to.

The person who coined the term "prompt injection" has the same gripe, because the original term genuinely did mean an attack using untrusted user input, a la SQL injection. But it's been conflated with jailbreak attacks in general, muddying the term.

Example of a bona fide prompt injection: white text in the background of a resume PDF, attacking a job application portal that uses LLMs to filter applicants. No privilege escalation is involved to give the candidate top marks on their resume screening.

Whereas a non-prompt injection jailbreak would be bypassing a safety filter, such as how Morse code might get past the filter and allow a user to request other people's cryptocurrency be transfered away. This is more akin to finding a poorly-secured, public facing API and then exploiting it.

I'm just trying to imagine this hypothetical company...

• They run AI agents without checking what it's doing
• They don't have backups or version control (or they've given AI access to delete it)

What else? Do they leave all their files in memory and only save at the end of the day to make sure a power outage could screw them over too?
It almost sounds like they want to lose their code.

Yeah, you need a local copy, an offline copy, and a copy in another physical location or you're not backed up.

Also funny in that issue:

The reporter "Ramon Batllet" (strongly doubt that is their real name, a search for it returns nothing but articles about this very issue) uses extremely polished corporate language and repeatedly uses "we" at first. Then when directly asked "Could you disclose on whose behalf you're discussing this?", they suddenly switch to "I" instead of "we" and claim to be a solo developer with no commercial interest. They still write in a style humans only produce for polished corporate reports, not like any regular human would actually do in a normal conversation.

So we have either a bot or someone very heavily leaning on bot usage for just about everything accusing someone of deceptive behavior, while in the same conversation trying to probably hide, but at least not fully disclose, their heavy usage of technology the accused explicitly does not want to interact with.

Yeah - Development and IT might feel slow, but there is a good reason why we've developed all those processes, access rights, approvals over the last decades. People are trying to burn down those "cumbersome" processes because they feel slow and AI promises them exactly that, but they will learn that everything is there for a reason, even that annoying SCRUM meeting

Where did you get that quote from? I can't find it in the linked article.

Really don't understand why people are so happy letting llm do everything for them.

I get a kick out of figuring out how things work, be it my car, a podman container running under nixos or flashing a price of hardware to gain further control over it. An understanding of action and concequence really gets skipped entirely.

People from my work happily let llms throw code together without any understanding of the how's and whys (I work in unrelated areas to programming / coding) and it just baffles me. Even one of my workplaces used a llm to make a Facebook ad, and so clearly it became an internal meme (and probably outside of work).

The Australian government is embracing llms and its baffling my mind as to why, ATO, ServiceNSW and probably another stack all gobble this shit up. Its in their training programs (images, text) and in your face while working. Actually absurd in my eyes.

it's not subversive. it's a string, it has no effect on the code output. Only a rogue bot would interpret it as anything except a string. No human user would ever encounter an issue.

any kind of undocumented subversive behaviour in your thing.

Fortunately, this behaviour is explicitly documented.

keep lickin' them boots baby. I want to see them shine!