Boob size

I'm almost 32 weeks and my boobs are only a tiny bit larger. A lot of people say you gain a couple cup sizes, but not a chance for me. Is this something I should be concerned about?