Technology

I promise streaming services and CDNs employ world-class experts in encoding

They don’t really care about the quality

It's funny that you are trying to make both these points at the same time.

You don't hire world class experts if you don't care about quality.

I have a hobby of doing re-encoding blurays to lower bitrates. And one thing that's pretty obvious is the world class experts who wrote the encoders in the first place have them overly tuned to omit data from dark areas of a scene to avoid wasting bits in that location. This is true of H265, VP9, and AV1. You have to specifically tune those encoders to push the encoder to spend more of it's bits on the dark area or you have to up the bitrate to absurd levels.

Where these encoders spend the bitrate in dark scenes is on any areas of light within the scene. That works great if you are looking at something like a tree with a lot of dark patches, but it really messes with a single light person with darkness everywhere. It just so happens that it's really easy to dump 2mbps on a torch in a hall and leave just 0.1mbps on the rest of the scene.

That will unarguably provide a drastically worse experience on a high-enough quality tv than a 40mbps+ bluray. Like, day and night in most scenes and even more in others.

I can tell you that this is simply false. And it's the same psuedo-scientific logic that someone trying to sell gold plated cables and FLAC encodings pushes.

Look, beyond just the darkness tuning problem that streaming services have, the other problem they have is a QOS. The way content is encoded for streaming just isn't ideal. When you say "they have to hit 14mpbs" the fact is that they are forcing themselves to do 14mbps throughout the entire video. The reason they do this is because they want to limit buffering as much as possible. It's a lot better experience to lower your resolution because you are constantly buffering. But that action makes it really hard to do good video optimizations on the encoder. Ever second of the video they are burning 14mb whether they need those 14mb or not. The way that'd deliver less data would be if they only averaged 14mbps rather than forcing it throughout. Allowing for 40mbps bursts when needed but then pushing everything else out at 1mbps saves on bandwidth. However, the end user doesn't know that the reason they just started buffering is because a high motion action scene is coming up (and netflix doesn't want to buffer for more than a few minutes).

The other point I'd make is that streaming companies simply have a pipeline that they shove all video through. And, because it's so generalized, these sorts of tradeoffs which make stuff look like a blocky mess happen. Sometimes that blocky mess is present in the source material (The streaming services aren't ripping the blurays themselves, they get it from the content providers who aren't necessarily sending in raws).

I say all this because you can absolutely get 4k and 1080p looking good at sub-bluray rates. I have a library filled with these re-encodes that look great because of my experience here. A decent amount of HD media can be encoded at 1 or 2mbps and look great. But you have to make tradeoffs that streaming companies won't make.

For the record, the way I do my encoding is a scene by scene encode using VMAF to adjust the quality rate with some custom software I built to do just that. I target a 95% VMAF which ends up looking just fantastic across media.