Seriously though, sometimes getting out of your comfort zone is good for you. Everything in moderation, even moderation.
Yes, the key here is to not let people push you around with this statement. Sometimes people like to say this when they implicitly mean “I am comfortable with X, therefore you should be comfortable with X.” This is not a genuine attempt to help you!
However, stepping outside of your comfort zone can also mean trying new things you were initially reluctant to try or hadn’t thought to try. Here you have the opportunity to discover some truly great new hobbies/interests that can enrich your life!