It's not a difficult concept, why do you think companies have HR departments? If your boss is making what he thinks is great bantz but is actually upsetting and a colleague goes to HR to let them know, it doesn't make the complaint less valid because you didn't do it yourself. Or if you feel that you have too much to lose so grudgingly put up with it, does that make the boss's behaviour ok?
Gender has nothing to do what I'm saying.