Help?

I’m currently 4 weeks 5 days pregnant, it’s my first pregnancy. I am so worried that having sex will cause problems, my partner is going insane as we haven’t had sex for a while now as I just haven’t been feeling upto it. Tonight he made me feel very comfortable and in the mood so we did it, it didn’t hurt as he was very gentle but I am so worried it is going to effect the baby. Can anyone reassure me? I’m due to go to the doctors for the first time next week.