Think of the atmosphere as a layer cake. The bottom of the cake is very dense and the higher up you go, these layers get less and less dense. Up on a mountain, the air is very thin.

In other words, there are fewer air molecules per cubic foot (volume of air). The molecules are farther apart and can hold less heat energy. Because "heat" is what we say when we mean molecules are moving around. The more they move, and the more molecules there are, the more heat you have.

It's also helpful to know that the air is heated by the ground and oceans. The heat from our sun mostly passes right through the atmosphere, not directly warming the air up very much. But the surface of the planet will warm up wherever the sun is shining on it. And in turn, the warm ground or the warm surface water then gradually warms the air from the bottom up. (This is because heat is transferred in different modes: radiant, convection, and conduction.)

And the warm air does indeed rise. As it rises, it gradually spreads out and cools off again. Some of the heat even radiates back out into space.

There are "fountains" of air constantly circulating throughout the atmosphere, and this creates weather patterns. It tends to snow on mountains because the warm air has carried some moisture with it on its way up. As it cools and thins, it can't carry the moisture any more, and the moisture precipitates out. Which is why we call it precipitation whenever it snows, rains, sleets, etc.

So by the time air reaches a high mountaintop, it's probably going to be cool or even frigid cold.

This is also why hotter regions, like the southern US, tend to get very humid in the summer. The warm air can carry a lot of moisture, and there is a lot of warm surface water. Our sweat is less efficient when the air is moist, because it takes longer to evaporate and carry the heat away with it.

Deserts have few water sources. So they also have hot dry air, and much less humidity, and therefore little to no precipitation. But they also get cold at night, because there's very little humidity to hold the heat overnight.

All of this is to illustrate the complex interactions between the sun, the atmosphere, and water (or lack of it) on the surface, and humidity in the air.

Inside an older building you're more likely to experience warmer air on higher floors than lower floors because the air is trapped in a nearly closed system and hot air rises. Of course, HVAC engineers try to compensate for this in modern buildings.

Others have covered the fact it's because of air pressure but haven't fully answered why that is the way it is.

It's simple really.

The force of gravity is also at play. As you go higher up, gravity gets weaker as you get farther from the earth's centre.

And it is that gravitational force that increases the air's density, same reason why if you keep going down in the water, the water gets denser.

For the heat to move around you need to be in a sort of goldilocks zone of density.

It needs to be dense enough that the fluid molecules can move around and spread the convection energy around... but not so dense they can't move much either.

Furthermore there's actually a couple different layers of our atmosphere.

First at our level is the troposphere, where heat is absorbed into the ground itself and radiated back out, as well as the perpetual heat from the earth's core, and reflected off the ground too (visible light).

The troposphere is warm and gets colder as you get farther away from the earth's surface, naturally. That heat is absorbed by the air itself so, as you get farther away it gets colder as it has more air to travel through.

Up higher is the Stratosphere, where it's ice cold and the air thins out.

However we get a sudden uptick in temp as we go even higher into what is called the Stratopause, back to briefly warm temperatures between the Stratosphere and the Mesosohere. Why? How?

Simple, this is the little sweet spot Ozone molecules hang out, forming a protective convenient bubble around the earth. Ozone absorbs Ultraviolet light from the sun and turns out that stuff is HOT, so there's a band of a hot zone right above and below the Ozone layer. Think of it as a toasty little bubble around us.

Above is the mesosphere which cools off again and gets back to being really frosty quickly, for the same reason the Stratosphere did, distance.

Then we hit the mesosphere, which is effectively the point when the atmosphere is so thin it stops protecting and is the "outside" of our protective blanket.

You can imagine this like earth being wrapped in a blanket, and the mesosphere is everything outside the blanket. Without any protection you are subject to the unbridled radiation of the sun which means you go back to being really toasty, as you get a bit higher you are effectively in space now and will soon enough hit temps that just cook you alive in a minute or two. Really bad sunburn zone.

So to answer the question overall:

Hot air rises... but only when there is air to rise.

Top of the mountains just don't have enough air anymore for it to really rise much more. It still does but the hot air rising effect just gets weaker and weaker as the air gets thinner due to less gravity.

Calvin's Dad: "Because the hot air rose all the way to space and isn't on the mountain anymore."

I saw a great one-liner, and two megalogs, but no Goldilocks-sized answer, so here's my attempt.

As air rises, the weight of air above it (all the way to space) is less, so it's less squashed, letting it expand.

It expands by pushing out on all the air around it, and every time an air molecule bumps a neighbouring bit of air away, but isn't bumped back so hard (so it expands), it loses a bit of energy - i.e. heat.

So as some air goes up, it expands and loses heat; or as it sinks, it squashes and gets more heat.

This is adiabatic expansion.

Appendix:

This might beg the question of why higher air isn't just heated by neighbouring expanding air, making up for its original loss. I think that can be answered by saying overall the top air is squashing the bottom air, so overall the top is cooler. Is that fully right? Right now I feel there's multiple ways to think about it and I can't write any clearly without long rambling!

Adiabatic temperature change

Fun fact: the temperature of space is actually thousands of degrees, but you would still freeze to death without protection.

(The actual answer is that atmospheric pressure is just as important as temperature in determining how "cold" something is)