Together for 2 months then became pregnant

I met my boyfriend in August. Things went really fast with us and we felt an instant connection. By the second week of me knowing him he had already met pretty much my entire family and I met his. Around 1 month of us being together we found out we were pregnant. We both have emotional issues from past relationships. He has a daughter that lives in another state and isn't allowed to see because the mother resents him despite his efforts to prove that he is a better man (he was much different in the past, sold drugs, etc. But now he's a personal trainer and goes to school). A couple months ago I was in a mentally/physically abusive relationship. I've expressed my doubts about being in a relationship before we were officially together. But just today he texts me that he's nervous and having regrets especially since I'm pregnant. I'm only 6 weeks but when I found out we were both excited and we even discussed moving in together and marriage. But now he's making it seem like this isn't what he wants. I guess my question is should I fight for a family or should I just focus on myself and the baby? I'm definitely having my child, I'm already developed so much love for him/her so I don't have time to waste my energy on a man that's comparing me to his last relationship.