taking on my partners son?

Ok so when I first met him he was upfront about having a baby and I took him on knowing that but I didn't realise how difficult it can be. His baby mama constantly rings him about irrelevant shit, uses the child as a weapon and even went as far as when he asked for the child to be dropped off after 3 because he wanted to come along with me to the hospital for tests and she starts shouting at him "she better not be pregnant" they were never in a relationship, just have a son together but what business is it of hers if we are trying? So 2 days before the scan which was his weekend of one night she rings asking if he can have him both nights which was no problem we love having him over for as long as we can and his mom was taking us to hospital so she can look after him for the 20 minutes we where there and she flips it saying he doesn't need to be going with me and he shouldn't be going now that he has his son. I personally don't see a problem. We've been talking about ttc but the whole situation at the minute is just putting me off, I can't be bothered with the whole drama coming along with it because she would be hurt apparently that he's with me through our pregnancy and we are together and their son didn't have that, not our problem she had her then new boyfriend playing daddy. I'm also afraid of the fact when he's older that I'll be doing what punishment I see suitable for my children and because my partner isn't allowed to tell him off and I'm not that he'll be throwing tantrums and getting off with what his siblings won't. I don't want any difference made and I love him and my so to pieces and would love a baby with him but this is just too much. Should I just get on with my life or hold out until he sorts out his shit with her?