Is it me or him?

So me and my boyfriend have been trying to have a baby. He has a little boy from a previous relationship that's already 2. My boyfriend I had in high school (3-4 years ago) we were also trying and no luck but after we broke up he had a boy. I'm too scared to find out if I can't get pregnant. I know my boyfriend smokes weed and drinks but does that really affect his sperm? Any advice??