My babies get bigger each time.

Has anyone ever had big babies then had a small baby? My first son was 9.5 and my second son was 10.5 lbs. People see me with my boys and assume dad must be big, which is not the case at all. I'm now pregnant with my third (unsure the gender) and I'm so scared of having another big baby. Everyone would tell me was so big with both of my pregnancies I'm so scared of this happening all over again. Has anyone had big babies then later had a average size baby?