Why am I not getting pregnant?

I'm starting to worry. I've been pregnant twice and have one earthside baby. She's 2 and we've been trying to get pregnant for 6 months with no luck. I've used OPKs, we have sex every other day and sometimes multiple days in a row, and I am healthy. I was vegetarian when I got pregnant the last time and had a healthy pregnancy and healthy baby. My family became vegan over a year ago and we eat very healthy. Our meals are well planned and my husband's blood tests from the hospital a couple months ago (he got sick and they thought he had meningitis) came back all within normal range. 
Now, my husband uses medical marijuana, and he was born with some extra stuff in his scrotom which supposedly would decrease his fertility but he's gotten me pregnant twice already. Is it possible that he is suddenly infertile now?
My OB thinks I'm fine because I'm having semi regular periods even though I'm still nursing our toddler. She thinks I'll probably get pregnant by fall but it's not looking good. 
I feel like I'm rambling but I just want to know if there's anything I can do to increase my chances of conceiving and if there's anything  I need to adjust in my diet.