Dating while pregnant..

I've been talking to this guy for awhile now and I really like him. He's taken me out to many places, he loves my son, wants to take me out of town etc. The thing is, I'm 21 weeks pregnant from a previous relationship. When we first started talking I was 14 weeks but as I get bigger, Im starting to feel weird about this. I always ask him what does he see in me, being that I'm pregnant by another man. He told me he doesn't care about that, and that it doesn't take away from who I am. I know he really likes me and he told me he lived me the other day but I don't know how this is going to play out if we actually be together. I don't want others to judge me and think I'm a whore, jumping from man to man. The guy im pregnant by now, we have a 2 year old as well, and the guy im dating says he also wants to take things slow and not jump into a relationship because he doesn't want to be hurt in the long run. I'm starting to fall in love with the guy im dating though and I want to be his girlfriend so badly. Should I just let this situation go?

P.S. The reason my previous relationship didn't work out was because he abused me physically and verbally for 3 years, also cheated with many different men and women, and I couldn't take it anymore so I left him.