I feel like "guys" is definitely colloquially gender-neutral in most contexts.
"Fireman" is clearly a patriarchic term that literally has "man" in it. In English "firefighter" is commonplace nowadays, but in my native Finnish, a lot of professions have "man" in the term, much in the vein of "policeman", "ombudsman", the Finnish equivalent of "janitor", roughly translated directly as "building/house-man".
We've replaced loads. Most of them are good. Some new terms feel natural and get taken into use, but replacing "man" with "person" rarely works for us without feeling incredibly awkward to use.
So my point is that we can reclaim those terms as gender neutral. Context matters. N-word being acceptable among black people is completely acceptable (and actually a very nice tool for emphasis when properly utilised), and it's even in songs without anyone accusing the artists of racism. (Well, for pop songs at least, no racist hillbilly songs made it to that level.) That being said, it definitely doesn't take away from it's power as a slur if someone uses it in such a way.
So I suggest we'll just use "guys, bro, dudes" as gender neutral and rely that people will understand from context when they're actually used to address men/exclude women etc.
Also, isn't "buddy" sort of neutral already? *goes to check* OoooooOoOooooh, it's from "brother" originally. Guess it's not as neutral originally.
Well that's s new one for me.