This is TMI but i want input

So im 19 weeks and 4 days pregnant by my my ex boyfriend. We broke up in november. (hes verbally abusive) and i haven't been having sex untill about last week. I started sleeping with my exhusband. (we dont have those feelings any more but we both are sexless and single normally) hes been pretty great cause me and him already have a daughter together. So he knows what im like pregnant and what i like. But this baby is not his and i feel like the baby knows the person im having sex with is not his father.... He kicks me a lot and kicks him. We have only actually had sex 3 times. But i feel like im invading my child. What your advice? Should i just stop having sex again? Is it ok to keep having sex?