RetroGaming

It doesn’t need to be the cheapest option, I usually aim for best bang for money. Thanks for any suggestions

Right now, in the current newish land of handhelds, the best for your money would be the Retroid Pocket 6. Here's their website link to the RP6's page if you want to check it out a bit more

It is a horizonal handheld, and honestly might be a bit of a higher price/performance than what you're looking for, but it will play PS2 uspcaled at around 3x - 3.5x resolution (that was my preferred sweet spot, and higher and there's no need for the AMOLED screen's output). It will play it at dead solid 60 FPS too.

Here's MGS3 upscaled to 3x on mine:

And GTA 4 with the same settings:

This handheld can also play...everything. All the older systems, PS2, Game Cube, some PS3 games, 3DS (if you're okay with the dual screens on one display), Switch (it is a MAGIC Switch system, with this screen it basically feels like a Switch Pro), and even some PC games with GameNative or similar. Super comfy to hold, beautifully perfect display, massive power, and with this one there's plenty of driver support too.

The only caveat is that the RAM crisis had Retroid drop it from 12GB RAM to 8GB RAM, but, I cover all of that in my review:

You can find my recent review of that one here with this link, if you're curious!

But if that is a bit more power and price than you'd like, the previous generation's Retroid Pocket 5 is still amazing today. Running the SD 865 means it has huge community support, and plays PS2 around 2x. You won't be playing any higher systems than that though!

Let me know if you have any further tweaks (would you prefer a vertical handheld? Or a certain price?)

(edit)

Retroid Pocket 6 color options:

Retroid Pocket 5 color options: