hand me downs?!?

Ok this question is for mamas that have boys and girls.  Is the fit of clothes actually made differently for boys and girls? I mean mostly for infant clothes since once they get older I'm pretty sure they do look very different, toddler and up. Im guessing obviously onesies and pjs are the same. Other then color and wording on the clothes is there an actual difference in the fit? Wondering if I should keep gender neutral clothes from my lil guy for my baby girl?