I want more kids, he doesn't.

Im in a long term relationship and when we first got together we had the kids conversation and he said he didn't want anymore (he had 3 from previous marriage). I was completely fine with that. Also i was diagnosed with PCOS a few months before we got together and was told i only had a 12% chance of conceiving and carrying to term. Well i got pregnant in December 2014 and i now have a handsome little 1 year old. My bf told me there was not going to be anymore after this one and once again i was fine with it. Then playing around one day i asked him about a 2nd child and he told me maybe in 5 years when we own a home. I was shocked that he said that and it kind of got my hopes up but now when i bring up in "in 5 years" he tells me no. No more kids. Now i can't help but feel like i want at least 1 more. Am i selfish? I understand he has other children and its a lot but even his aunt says he's not being fair to me. In the last few months the family has had 2 little ones and its given me crazy baby fever. I just want to keep all the babies every time they come around!