Sex during pregnancy

Ok so me and my husband have sex, maybe 3 times a week or more..I'm 23 weeks now. And he always comes inside of me, his sperm inside... I remember the doctor telling me to have protected sex with condom, I just didn't understand why?? Is the sperm bad???