Are we going too quick?

Me and my bf have been dating for about 4 months but have been knowing each other for about 6. We are in love or so it feels as if we are. I have the mentality of whatever happens,  happens. Only problem is he already has two kids, an almost 2 yr old and a 7 month old, whom i adore so much. They are starting to get used to me since i babysit them once a week and see me 2 times a week. He had also recently gotten out of a 2yr relationship with his babymomma before his second baby was born but his babymomma is still in love with him. She was unfaithful in the relationship which caused them to seperate but wants him so badly. She knows I'm in the picture now but he still hides me from her, saying he is only avoiding drama with her and she doesnt need to know his business, but that makes me feel as if im the "side chick". He says he does not want her and is in love with me and wants to be with me and I shouldn't worry about anything. But im not all that worried about that because i already met his close family and currently, slowly meeting the rest of his fam. So i know he is taking us seriously. 
​I already see him struggling supporting his babies, and all the stress coming being seperate from the mother. And i am now currently panicking if im carrying his third child. I personally wouldnt mind since it would be my first and i know he will support me in any decision i make, but i also think of him and i dont want him to go through what he is going through now and stress over financially supporting a third child. I believe its too soon for everything but it feels right but i know its wrong. 
If i am pregnant, i would like to keep it but i know it would be best to not?. I dont know what to do. He says who cares about the stress and what people think, that we would figure something out if i am pregnant.