The problem is that they are happy to let the building sit empty most of the time

The value of the land and building continue to go up as an investment, even if they aren’t earning money today on the space .

So they don’t actually give a fuck if it sits empty, but society does 

I'm with you, but you're kidding yourself if you think the billionaire is going to suffer. They have leveraged the value with banks, and would skip on down the road with their fortunes while the banks that make mortgage loans have to shore up their books at the expense of common folks. Homebuyers, small businesses, and taxapyers will be expected to cover the losses.