I have an alias I call "upd" that runs "yay ; flatpak update", I just run that, press Y at the first prompts and then let it run in the background while I do other work. It really doesn't matter at all how long it takes. I do have NVidia but generally I don't feel it takes very long as we don't get new kernels every day. You could use the linux-lts kernel for much more rare kernel updates.
It's a bit like bittorrents, I don't need them to download in 30sec, I start it and return to check on it whenever I think of it.
I have changed my opinion on flatpak btw, I really like that the apps are not spread all over my system but instead sandboxed neatly, have fewer dependency versioning issues and it's really easy to use.
That's abnormal, it shouldn't be like that. My flatpak rarely has updates (compared to Arch/yay) and they're quite fast, still less than a minute even if there's updates to the NV libs (I didn't time it). There must be some kind of particular issue? What's your setup?
Looking at it - I got flatseal, chrome, firefox, thunderbird, dropbox, steam, joplin, cryptomator, mesa, NV libs, gimp, discord, resynthesizer, libreoffice and some other bits on flatpak. It's on an SSD, Internet 150Mbps. Is it installation or download that's slow for you? With it being 20min I would guess there's a problem with the download speed from the server, routing issue to flathub, etc? Flatpak is not that much of a slog.