Does size matter? (TMI warning)

Sᴀʀᴀʜ • 💜♐️sun💜♎️moon💜♒️rising💜
So this is a little akward, but my husband is maybe about 4 in, and we are trying to get pregnant, I was just wondering if size really matters when it comes to fertility? Should we just keep trying or go to the doctor?