Frustrated - family issues

So I've been married for 7 yrs but due to immigration issues with my husband we were apart the first 3 yrs but I'd go visit him as much as I could. Well now we are both 38 & 39. I've been desperately wanting a family but our first 2 yrs together were not easy...we had issues and didn't have much sex to try to make a baby. Last summer we finally started trying actively but he's like a roller coaster. ..some months we try and some we don't. It's gotten to the point that we are seeing a fertility specialist now. My problem is not so much that that I want to vent about though. My issue is that he has his mom sister (who is 42 and a widow) and her 2 kids both under 15. He wants them all to move in with us! I know if I say no my marriage will be over as he feels he can't leave them alone back home. They live in an unsafe 3rd world country where now a days even schools are being attacked. So it's hard for me to say no but I also feel I have never lived a normal married life yet. I just don't know what to do. The other night I lost it on him and broke down in tears feeling lost and torn. I wanted to beat him so bad like why did you get married when you know you have all this responsibility in your hands. Also he has a older married brother in UK who does not contribute one bit. When I bring him up its like I'm going into forbidden territory. Any advice. I could also end everything my marriage and move back home and live a comfortable and happy life but that's not in me as I want my marriage to work. BTW I'm from the states and my husband is from south Asia and we are living in canada for now.