Relationship advice/ Second baby

So my boyfriend and I have been together for 4 years we live together but have agreed that we don’t want to get married. We have 1 son together and discussed having a second child as I am an only child and hated it! He knows how I feel about being an only child. We had discussed having 2 and he also said he wanted 2. Well we had been trying for the last 4 months for another baby (our son is 28 months) and then all of a sudden he says that he doesn’t know if he wants anymore kids, and says he’s fine with the one we have. Of course I love my son and am fine if i can’t have anymore children but to just not try is very disappointing to me.

So I guess my question is, would this be a deal breaker for you? Our relationship isn’t the most amazing, he can be a bit arrogant sometimes and is high maintenance. He’s a good dad but he also prioritizes other things other than me ..

I’m just super confused.. I told him he needs to tell me ASAP if he wants more children because that could be a deal breaker for me.