I feel like giving up

Our relationship has been more than rocky these past few months and last night I wanted to walk away so bad. I feel like he's forcing me to leave while still begging me to stay.

We've been together for over a year and a half now, between 4 kids, 2 pets, and living separately for the past 7 months, and TTC... it's been very stressful to say the least. We've been talking (official) marriage and it seems like some far off dream lately. Last night we argued about him moving back in. He moved out to help his brother get an apartment, in that time the mother of his kids needed a place to stay. 2 of our girls are from his previous relationship and their mom does not like me one bit. She's honestly what started our argument. She feels since he wants to be with me, she's going to go for full custody and not allow either of us to see the girls. I love those two like they're my own, practically raised his oldest since she was 7 months (she's 2 now). I told him I would never say to put me before the girls, but you shouldn't invalidate my feelings to make the mother of your children to feel at ease. I just feel like he's pushing me away to please her, but he says he's not and he just wants to make sure his daughters are taken care of and their mother is stable. Being a mother myself I know that you should ALWAYS put your children first, but it seems to me that he's putting their mother first. She could always move in with one of her family members or friends in my opinion. Or even just stay there while he lives with me.

Am wrong for feeling how I feel? Am I wrong for wanting a less complicated relationship and expecting my man, the man I call my husband, to back me up?