LetsEncrypt provides free certificates. I would setup Nginx Proxy Manager and use DNS challenge with your dyndns provider to get HTTPS on your home services.
My problem - and I'm not alone - is that I really don't want to expose anything publicly. Is there a way to do this without exposing anything to the Internet?
You don't have to expose Nginx publicly. It can exist privately on your network. I have my own domain and DNS server internally. For example nginx.home.datallboy.com and jellyfin.home.datallboy.com will resolve to NPM server at 192.168.1.10. Then nginx can listen for jellyfin.home.datallboy.com, and proxy those connections to my Jellyfin VM at 192.168.1.20.
Since I own my domain (datallboy.com), I let Nginx Proxy Manager do DNS challenge which is only used to authenticate that I own the domain. This will insert a TXT record on public DNS records for verification, and it can be removed afterwards. LetsEncrypt will then issue a certificate for https://jellyfin.home.datallboy.com which I can only access locally on my network since it only resolves to private IP addresses. The only thing "exposed" is that LetsEncrypt issued a certificate to your domain, which isn't accessible to the internet anyways.
You do not have to create your own CA server.
All things selfhosting and homelab related Resources: - https://github.com/awesome-selfhosted/awesome-selfhosted - https://github.com/awesome-foss/awesome-sysadmin