Breast growth and breastfeeding?

I always hear women talk about how much their breasts grew when pregnant, but mine have barely grown in size. I do have issues producing enough milk and I’m curious if this has any correlation? I’m currently 31 weeks pregnant and I’m trying to prepare. This is my third pregnancy and suggested my last two I could not produce enough to give only breastmilk and the last two are 11 years apart. I was thinking how my breasts never got bigger and if that’s the issue. Last time I tried everything but ended up having to supplement because I couldn’t produce enough. I ended up stopping at 7 months after supplementing with formula. I always gave the breast first and pumped after.