I have a lot of indirect experience/knowledge with caves. I don't have to be one of the people who directly explored one to know one of the first things one learns about caves is how ill-suited for wandering around they are. Slopes can change on a dime, it's incredibly rocky inside, and they fill with whatever falls from the sky. Imagine hiding in one of these, as someone with less intelligence than us, and not expecting to even stub a toe, let alone fall or suffocate. I assume it must be a concern because all the cave hobbyists I know (even if I only know a couple) say they have to split up based on their physical skills (so one chooses a slope-free while the other a rain-free one as would be the case for me as someone who never learned), though I admit I'd be amused if cavepeople didn't ignore these slopey checkpoints and instead it caused them to make some inventions.
Another thing that sticks out to me is the rule against fire. When cave explorers (not me) venture in, the most important rule of all time they learn is that it's an absolute cardinal sin to light so much as a match in a cave, let alone a torch (in contrast to Indiana Jones movies where that's the first darn thing he does). The heat from fire is enough to interfere with the stone composition, which in turn threatens to collapse a cave. Imagine having just discovered fire, and you go running to show your family but everyone dies before you can say "hey Mario look what I made". I wouldn't expect a caveman to know about that rule, but I would expect them to feel tempted to find out the hard way.
Finally, there's the fact they're filled with disease. Most notably from the cave animals; while things like ticks and rabies are not common in cave animals, they do happen. If that wasn't enough to outright stigmatize dwelling in a cave, even the environment itself is viral. There are caves where the reservoirs are like 100% condensed bacteria. That's got to send awful mixed messages to seek refuge from an oasis and suddenly you have a fear of water like me.
How did caves become such a go-to and one where nobody is depicted as having any serious accidents in?
Africa’s climate doesn’t offer that much deadly cold.
And where caves were involved, they may not have been a constant dwelling but a place to retreat to in times of need.
Your spelunking friends don’t consider rain a deadly threat because they have REI jackets and access to antibiotics, so their attention is entirely on the issues of rain destabilizing the cave. But disease and exposure were immediate deadly threats to more primitive people.
So hiding in a cave during the rain, despite your spelunker friends’ modern safety standards, was probably a survival tactic.
I mean, there were cave dwellers in Europe and Asia too. The richest cave culture finds were in France.
I can attest all threats are considered to the best of one's ability (minus the "things everyone is willing to risk"), even with everyone's REI vests/jackets, which is why everyone often takes different routes based on what they're good at (for example, one is bad with slopes, the other panics at puddles, though they insist this is "fun"). Once a cave even had classic stereotypical radiation in it (I should note one unspoken con of caves is they have an extremely high radon composition, which is natural in caves even though hearing it might be a mind changer). Every cave is different.