Is it time to give up? ** very long but please help***

I need some serious advice. I've been with my husband (who has 2 children from his last marriage) for 4 years and for the past year he has been talking about wanting more kids. Well after 3 years of trying we decided I needed to see a Dr about why we've had no results. So I've been put through every test my Dr can think of, I'm on too many medications to count. All so I can give him what he wants. I found this app and have been using it so I can actually show him when we need to try, and how much I'm trying to make this work. But when my fertile windows come he's just like oh I don't feel like it now. But after my time is up I catch hell for the negative tests. I'm so lost. I've tried talking to him but he beats around the bush and starts an argument. I'm starting to wonder if I even want this anymore. Oh and did I mention that the whole time we've been together he's smoked or gambled away all our money. We've been homeless for 3 years. I try to save bits and pieces but he always manages to find my money and bye bye it goes. Every time I try to explain to him I don't want this life he flips out and tells me its my fault for not stopping him from spending money. I've told him I won't have a child drug through this life. He says oh in 9 months a lot can happen. He says I'm crazy and need pills because I keep "nagging" him about money, and food and maybe a home. I sit in bed and just cry. I love him and I love my step children more but since we move away from them he really won't even put in the effort to go see them. He's a great dad when he's with them but to me that's not enough he needs to try more. I don't know if I even want to stay with him let alone give him a child to "torment" with this shit.

Am I really crazy and just need to relax? Or is there something not right about this?