I'm no doctor but I believe it is because the sex of a person isn't determined until late on in the pregnancy and all the bits are in place before this happens.
Either that or nature's way of making it easier for a sex change to take place without the need for a nipple transplant.