Long vent and critique, should i leave him?

So my so lives with me we have been together a year and 2 months and i have a son, who he willingly calls his own. (His real father helped plan the pregnancy IT WAS HIS IDEA TO BEGIN WITH, but he left shortly after the birth and never came back because he got another girl pregnant and i didn't want a relationship with him, how this justified abandoning his own flesh and blood i will never know) so my man has since taken over and been a great father to my son. But lately, idk.. I just dont feel the love anymore. And this is by no means his fault. I just dont feel in love with him anymore. Before i had time to miss him. But he's since moved in and it gets rough we argue alot and lately we've had a good time no real arguing or anything. But i lost so much love because even though now he realizes he said some harsh things i cant forgive him. Overall a good man, voluntary father, hard working, not a cheater (and i have his phone all the time) he just said something he claimed he didn't mean during an argument about a fortnight ago and i cant let it go. He loves my son and i love that, but every time we argue i feel if it weren't for my son he would leave. Everytime i try to kick him out he asks about the baby and if im taking him away from him and he knows it makes me sad because i never knew my father and the fact my son cant see his real dad makes me angry (that relationship alone took a toll on me) and now, i feel like if i leave him, im taking a great father figure away from my son. And it breaks my heart. But he blows up too quickly on me. We were engaged and since that argument i have called it off and don't wear my ring. Like i said LATELY he's been an angel. But idk, i just cant see past his words because i feel he meant them. He yelled?. " i dont care about you, i dont care bout what happens to this relationship, fuck us, fuck you" all because of something so minor that i cant even remember the reason. Before this we had a bad arguing rut., (like a daily screaming match because he blew up over miniscule things and called me a piece of shit and overall kept calling me lazy) Which is nuts to me because i attend school work and parent, and clean my house. He just said it to spite me. But he never went so far as to make me feel our relationship just doesn't matter to him anymore. At this point i just don't let it go. I have always hated those type of people who held grudges because it can hurt your so. But idk. Am i justified in y'all opinion to leave him? Am i overthinking? What would you do in this situation. Please be polite about your comment. I think i just need guidance or another woman's words right now. I'm posting anon because i dont want him to come across it in my phone and start a screaming match considering its been good about two weeks now.