Is it genetic?

My mum had some trouble TTC, and my sister also got told she'd never have children's (God bless she had 2 heatlhy boys) and my brother had also been told that he has a low sperm count and has a slim chance of having children, I'm worried it's effecting me and I'm scared of going to the doctors because I can't face what they might say to me.. Came off my pill about 5/6 months ago, I know it takes a few months to come out of your system anyway, am I just been paranoid?