Technology

You need a person with a lot of experience to get something useful from this bot,

Not entirely true. You get a lot more useful things from the bots when they are driven with people with a lot of experience. The problem that's coming now is a magnified version of the "skript kiddiez" from early Google days where inexperienced people could just find exploits on the web and copy-paste them. Today, the LLMs actually can find vulns and develop exploits for people who don't have any knowledge of the languages the exploits are being written in.

every time we actually measure, the results that your experienced person will be quicker and better not using it at all, and doing the same work themselves.

From my perspective, your data is out of date. I've been tracking the "usefulness" of frontier models in accelerating development speed for experienced people over the past 2 years. Two years ago, total waste of time. One year ago - equivocal, sometimes it accelerates an implementation, sometimes not. Six months ago, it was clearly helping more than hurting in most cases, and it has only continued to improve since then.

Knowing what you are doing helps. Trusting that the LLM will help, helps - if you set out to show it's a waste of time, a waste of time it will be. Lately, treating the LLM like a consultant, just hired, likely to disappear any day, helps. Take the time to run all the formal processes, develop the requirements documentation, tests, etc. Yes, that "slows things down" but not in the long run across realistic project life cycles - even with humans doing the work. Also along those lines: keep designs modular, with modules of reasonable complexity - monolithic monster blocks of logic don't maintain well for people either. LLM implementations start falling apart when their effective context windows get exceeded (and, in truth, people do too.)