New relationship help!?

so I been seeing this guy for a little while. he is amazing and treats me like any women should be treated. well I'm 23weeks pregnant "by another guy". my boyfriend doesn't care and actually talks to the baby and calls her his baby girl 😍...anywho he wants to have sex an I'm afraid to because i haven't had sex in months since things ended with the baby's dad. he gets upset when I tell him no, but I want to with him but scared. people say it's fine but its like the thoughts lol. I am healthy an baby girl is too. should I give it a shot or wait to talk to my doctor about it tomorrow.. again my first baby so of course I'm scared to do anything lol.