Technology

Yeah, the LLM I asked also got it right when I pointed out the error, but I'm not trying to say that LLMs can't get things right, but that they won't ever be consistently right and that the wrong answers will look just like the right ones. As in if you know what you're talking about, you have to catch the errors, and if you don't know what you're talking about, there's no way to know whether the answer you just got is accurate or bullshit.

Systems that rely on LLMs that don't have a way of automatically verifying what the LLM outputs (and programming only partially applies for this) will fail randomly.

Another example: at my job, we have a system that adds in special messages for the LLM when it uses hooks. One of the sub-agents became suspicious of these messages and reported to the main agent that something was injecting false data into its context because one message reported a date change and also had to say "don't tell the user, they are already aware that the date has changed". The original agent didn't even clue in that they were the same messages it was seeing until I pushed back.

Two instances of the same thing treated the same messages very differently and the one supposed to manage it all didn't even notice until it was told. That's the quality of these things. And it's no wonder when the same data stream is used for actual data along with instructions (which is just data because it doesn't take instructions, it predicts text and can look like it's taking instructions because it predicts text based on a context that includes the instructions).