jj4211

I saw a headline where one of the big AI people said AI could do everything better than a human... Except make decisions. So the "executive" class is the only class that is "fine" by the logic.

I don't think they half the self awareness to realize they are in fact frequently pretty pointless even ignoring AI.

Curl guy has written about a couple of these stupid CVEs, for example: https://daniel.haxx.se/blog/2023/09/05/bogus-cve-follow-ups/

One I recall was that if you asked curl to write out c code example of libcurl usage, you could get it to write out arbitrary code of your choosing. Note that this required you to have write permission and curl and then with your malicious c code, you then had to compile it and make it executable and run it yourself. So a very roundabout way to use curl as a text editor, and they considered it an arbitrary code execution issue, despite not actually executing the code.

Yeah, CVEs are usually nothing when you get down to understanding, especially kernel CVEs, which almost always declares a CVE for almost any bug, because it is easier that trying to think if it is a security issue or not and basically just assume it could be.

Huge pain as in my work we have a security policy where any unpatched CVEs that cannot be updated away must have a fairly significant writeup delving into the nuance of the CVE and what mitigation has been applied or a rationalization of why it isn't a risk and by policy we have to second guess every CVE assessment from our vendor, who we explicitly pay to triage and fix this stuff so we don't have to... They used to at least allow us a pass on "low severity" (that's still pretty flawed), but they decided that didn't sound "tough" enough and now every single one must have an answer. So every month a few people have to spend a few days just reading tons of CVEs that are not yet (and frequently never will be) patched by vendor and rationalize it away for the security team. Sometimes the security team will get odd and demand we build our own from upstream (most recently, vim of all things we were mandated to build from source).

The content on Internet will invariably biased towards the novelty. Between that bias and enormous marketing spin and the most self important people gravitating towards it, it's an expected reality. One that will be applicable to some scenarios.

Content saying that for some situations, the existing methods remain best isn't going to light the world on fire. Also, tech folks tend to be more shy about "not getting" a seemingly great new tech.

In terms of how much it applies to an individual situation, it's too nuanced to really make a universal judgement. But in my local circle where I can grasp the nuance, the teams that have gone all in on deeply agentic are teams that already were kind of crappy.

I have both used agents and have been the "victim" of heavy agent users.

If you can provide an utterly well bounded problem with perfectly verifiable criteria for it to retry against until the tests pass and the tests are full and valid for the use case, it can work. Similarly, if your code's reality is forgiving and flexible, and a 'close enough' result is good enough to get the human software user in the right ball park, you might be able to extract decent behavior.

However, if there is a means by which the answer can 'look correct' yet be wrong in the real world and the real world scenarios require accuracy and precision, there's huge gaps.

Especially if the agents control the coding and the test case generation, seen plenty of times where it talked itself out of a test case that was failing when the test case was in fact correctly showing a flaw.

A number of people have shared their experience with you and you seem to be unwilling to acknowledge that your assertion of universal slopping it up as reality is not utterly universal.

Frankly, your use case sounds like something has gone horribly wrong from the outset. If something needs several thousand regex patterns, then something is very very wrong, and there's zero chance you have a comprehensive solution as it stands. Either trying to use regex for a use case that some natively AI approach might actually be warranted for, or regex is being used poorly and each one is too limited, or some other approach entirely is warranted.

Perhaps your experience is the exception?

The problem is that every time, in the moment, people advocate and then when it comes up short, come back with "Oh, you did XYZ 3.2? Yeah that's busted, 3.3 can really do it" and rinse and repeat and it's hard to take that argument credibly when it's a treadmill of dissing yesterday's tech but swearing today's is different.

The current best of breed I have very limited exposure to (too rich for my blood), but it didn't seem overwhelmingly a slam dunk in my interaction. Incremental value going from more modest models to those don't seem to justify the price tag.

And every developer that is merely curating fully agentic workflow I have dealt with has pretty shit functionality and no idea what the hell they are doing. Whatever the rhetoric is, the results are just utter shit. Somewhat better if the problem domain can be infinitely retried without consequence with results that can be perfectly verified to let it drive eternal retries (while burning through token budget), but generally I just see pretty shit software.

On the other hand, a lot of these groups that I say has pretty shit AI software formerly had pretty shit normal software. Problem being clueless management now thinking they must be smart because they say things more aligned to the AI hype.

Agents can already write much more organized code than any developer in my company.

Then you must have nothing but absolute shit developers.

Been suffering the hacks and downtimes for over two decades waiting for the day when corpos come around on this.

Hard to continue holding out hope.

The big problem is that sometimes it's "good" at diagnosing issues and sometimes it's very bad and in both cases it looks the same if you don't have a way to know yourself.

I've seen some of the fodder people consider 'well formed playbooks' and it's also a bit of a crapshoot there. It can speed up some tedium, but so many users get in over their heads.

Today had a junior proudly provide an untested playbook that just did everything wrong and even if it did what it purported to do, it would have been hard coded to certain things that are not what we want to do.