My boyfriend is not excited about the baby because it's my ex's

So my boyfriend and I were together for almost a year and we broke up because his ex-fuck buddy/current friend told him that her friend who worked with me at the time saw me kiss some guy at work. It wasn't true the name and description of the guy I apparently kissed didn't even match anyone I knew at work he literally didn't exist anyway, I went to his house as planned and when I tried to go inside he stepped in front of me and asked who "Whatshisname with gauges?" was? I had no idea what he was talking about I said "I don't know??" And he told me how one of my coworkers who is friends with his ex-fuckbuddy called her and told her I kissed this made up guy and the fuckbuddy told my boyfriend I told him that wasn't true and there wasn't anyone like that at my work anyway and he said he believed his ex-fuckbuddy over me so we broke up and I drove home. The next few months after I got with another guy and we only dated for a few months and I broke up with him because he turned out to be controlling and abusive. Me and the first guy started talking as friends again and ended up getting back together and worked everything out he doesn't talk to the ex-fuckbuddy or see her anymore. The thing is a few weeks after we got back together I found out I was a few weeks pregnant with the abusive guy's baby. I told my current boyfriend and told him he did not have to feel pressured to stay with me and I would understand if he wanted to leave me (I had already decided I wasn't going to tell the birth father about the baby because I'm terrified of him abusing the baby and he lives with an illegal who smokes pot all day long with his kids in the house and doesn't pay attention to the kids so please don't judge me about not telling the birth father) anyway my boyfriend decided he wanted to stay with me and at first he was good about the pregnancy but apparently that's only because he thought there was a slim chance it was his but once he realize it wasn't because of how far along I was he is distant and cold when it comes to the baby. He's not involved at all, he hasn't gone to any appointments with me, and everytime I talk about the baby he looks/acts annoyed and cuts me of while sighing and saying "Yeahh.." Then changes the subject? I don't know what to do to get him involved I don't want to force him but it makes me feel like I'm going through this alone. I know he loves me and I love him too but I want him to love the baby not just me. It's obviously not ideal for me to be carrying my ex's baby but there's nothing I can do about it I just wish he was more excited...