Big baby at birth

One thing I'm worried about is having a big baby...i know it's crazy thing to worry about, does the thing that says what you weighed as a baby and what your SO weighed as a baby will depend on what your baby will weigh hold any truth? I know the ultrasounds are difficult to judge weight but at 32 weeks they told me he was 4 pounds... please give me some clarity lol

Btw I was 5.5 and he was 8.14