Often it’s because I’m just too afraid because the person is ridiculously good looking that I don’t think I have a chance, rejection always sucks. I’ve been with an attractive person or two in my time, so I know it’s not impossible, I’m ok-ish on a good day, but I know I’ll be so… distracted by their good looks that I probably won’t have anything intelligent to say.
There’s also the chance that either of us is in a relationship or workmates, which just isn’t a good mix and eventually leads to problems. Otherwise, I wish casual sex was just a common thing that people just did without any stigma attached to it (assuming STDs weren’t so rampant and safe sex was also just as common).