Too early?

I found out at 18 weeks I was having a boy. I have bought him a blanket and an outfit but everyone seems to say its to early to start buying him things because miscarriages can happen or whatever. Should I wait or go ahead a buy him what we want? I am 21 weeks now.