He Changed His Mind.. Sort Of

Ha
So back in December I had gotten sick, it was just some weird cold I guess. I was having lower abdominal pains and bad smells were really strong, I couldn't eat certain foods because they smelled or tasted bad. I was nauseous. A lot of symptoms pointing to pregnancy. I was on a mirena so I couldn't be pregnant but I waited for my period and it was late (probably placebo) but it came. I was still sick tho.
My boyfriend was the one who suggested pregnancy. He was the one who amped us both up thinking I could be pregnant despite my period eventually arriving. His reasoning being his exwife still got her period. Her pregnancy tests showed up negative even 4 months in. Blah blah blah. She has an ovarian cyst problem tho which I've heard turns out wrong pregnancy tests. 
Now a little back information is that between the two of us we have 4 children (3 full-time and 1 half-time). So despite his admittance of always wanting a second child he was satisfied with the baby I brought into the relationship and loves all of our children equally. Soon he was saying we have enough children we don't need another.
Back to December, he starts saying things like "I'm kind of excited now", "What happens, happens", "I hope we have another girl". I knew in my head this can't be a pregnancy I'm on the mirena but he had me so excited. Just the idea of me being pregnant made him so happy and he changed his tune about having enough children so fast. But I had blood tests done (since I was due for them anyway) and they just added in a pregnancy test for me and it came back negative. I even had X-rays done to find out what was causing my stomach pains (which turns out was stress). 
I held it in for days but finally one day I couldn't take it. He had dropped me off at work so he could run errands and pick the oldest up from  school. I wouldn't look at him. Some tears had escaped and he just left me alone because we didn't have time for it yet. When he picked me up it was dark so I hid in my hood and cried on the way home. I sucked it up to put the kids to bed, took a shower, and then went to bed myself and cried. He came to bed and pushed me to open up to him until I admitted that I still want another baby. I wasn't done either before we got together but that didn't change for me. He heard me out and said make the appointment to get my mirena removed. He said we would make it work financially, and he was upset when we found out I wasn't pregnant too. So I did.
It's been 2.5 months since I've had it removed and about a week ago he changed his mind again. His ex wife announced she was pregnant about a month back and he started day dreaming about a little baby girl again and asking what I'd think we'd have and I started getting upset because I thought he was just going to go through with it to throw it in her face. But suddenly there's no reason for it anymore/again. I feel like I've pushed him into something he really didn't want and I'm confused about whether or not I should continue to go through with this. When I asked if he wanted to use condoms he said no, they don't feel good and again, whatever happens, happens.