Hard choice need advice!!

Yunuel
Ok so tonight when I was at work my fiancé and me where texting back and forth like we usually do on our graveyard shift that we work, in the process of us texting back and forth the subject of me stopping TTC came up, I had mention to him that we should stop for now until all our bills and things get situated and his whole baby momma court thing gets settled down (FYI he has kids and his baby momma took him to court to get full custody of the kids, when he has had his kids the past 3yrs and she was no where in their life's, she has been living her life with her new boo without caring for her kids) because it was becoming to stress on me that I had to be doing it myself while he pays for a lawyer and the house payment only, yet I have had to pay for food, clothes, bills, gas and everything else, anyways I was telling him about it and he answered me that "what if later on he doesn't want anymore and when I was finally ready, he wouldn't do it".... knowing that crushed my dreams of being a mother, I mean I love him and see myself with him for the rest of my life, but I just don't know what to think! Should I just settle with his kids and stop TTC for god, or should I do it now??? It is hard for me to get pregnant due to my PCOS and Endometriosis. 
Please help, any advice will help at this point!