technology

Youre right about quality but the problem isn't just code quality, its also the sheer volume of code and its unmaintainability. Quality has always been pretty low in commercial software, but there were practical limits to how much of it could be created, and with enough labor thrown at testing and maintenance, as you say it clanked on.

When it's easier to just generate a whole new page in the UI than fix bugs or make usability improvements though, when it's easier to generate a whole new separate app than to introduce a new feature into the existing one, and when completely nontechnical people are running around doing this stuff without understanding the consequences, risks, etc. I think its very fair to describe it as a dark age. Large parts of the industry and the world are already there in the thick of it, and it is bad.

Social structures will adapt but that won't bring back the old way of doing things or reduce reliance on AI, if anything it will entrench it further, until/unless costs become prohibitive

I don't know how things in the industry are now, but back in my day you didn't deploy a Microsoft product until the second service pack had been released. Trusting software to be reliable and secure was always a no no.

This notion of “quality of output” misses something important though, which I think our economic system predisposes us to ignore.

I think most of us have this concept that the function signature of “working on code” is something like:

implement(spec): CodeChanges;

But I think this hides some stateful side-effects. There’s a monadic thing going on here, and I think it’s really more like:

implement(knowledge, spec): [CodeChanges, KnowledgeBundle];

…where KnowledgeBundle is a whole bunch of things: latent info about how things currently work, obviously, but also the yet-unnamed missing abstractions you noticed, the existing named abstractions that are starting to strain — basically the sense of the trajectory of the codebase.

Even really shitty programmers get some sense of the trajectory of the codebase, but this is something that even the best frontier models fail at.

Tangent:

I saw a pretty cool visualization of Kalman filters the other day (substantially AI-generated, coincidentally), and I thought it matched something about this dynamic.

• Whole video: https://youtube.com/watch?v=zFmO_b05jU4
• Short clip: https://youtube.com/shorts/vMRrWb_lf84

If you’re only concerned about the “next step”, aka “quality of output”, your error will accumulate and you’ll drift over time. You need to be able to evaluate where you are, so you can correct that error and stay on course. And to do that, you need to be present inside of the code and feel where the pain points are, or your estimation will be so far off that it’s useless.

I think our economic focus on “output” and our obliviousness to internal states (or even denial that they exist at all) is going to make it very easy to fall into the trap of taking a long series of next steps that individually look very sensible, but over time accumulate unmitigated error and drift towards disaster.

Pareto principle gets even worse for LLM code, the Photoshop clone won't exceed Krita in quality as time goes on