What are everyone's thoughts about paying off your mortgage pre-retirement?

I got enormously lucky and refinanced at the very bottom of mortgage rates in 2020. Conventional wisdom would be to never pay this off as it's effectively free money. However it is costing us roughly $20k a year in added expenses, which I could see as having a massive impact on ACA and FAFSA (assuming a future child) in retirement.

I'm getting to the point where if I want to pay this off before retirement, without having to sell investments, I'd need to start diverting funds from investments into a cash account (since HYSA is currently > mortgage rate) to have enough saved to lump sum in at retirement. Another option I've been considering is saving enough cash/equivalents to cover mortgage payments without paying it off. However I'm looking at roughly 20 years of remaining payments at retirement and I'm not sure if that is a good strategy or not. Thoughts?