No luck on a baby!

So I’m 24 and have 3 children. I went through a rough patch in life and adopted them out to the same family. I’ve finally gotten my life together and am ready for a baby with my husband. We have been together 2 and a half years and married about 4 months. We just started about 4 months ago trying and I can’t seem to get pregnant. I’m so worried it’s him and don’t know what to do. We were talking about seeing a fertility doctor. He has no kids but does smoke weed. I’m worried maybe it’s lowering his sperm count and need opinions. I wish he wouldn’t smoke but idk what he will say. He always has and I knew that when we got together. Please don’t judge me. I would love advice. My in laws want a grand baby so bad.......