Yes! You just have to set up your reverse proxy to send everything through it and it'll block the unauthenticated access.
The downside is that apps stop working since they don't have a way to authenticate with authelia. I've installed it as a PWA on my phone and use an old laptop with the TV interface on my TV, but it's not perfect
Both jellyfin and authelia support reverse proxies.
Here's jellyfin's guide: https://jellyfin.org/docs/general/post-install/networking/reverse-proxy/
And here's authelia's: https://www.authelia.com/integration/proxies/introduction/
There's some restrictions (like websocket support) but it's not too bad to set up.
Still, if you don't need to expose it to the internet, put it behind a vpn.