Sex and early pregnancy

So of course I'm paranoid to do anything that might harm my tiny baby. I'm 4 weeks and want to know is having sex going to harm anything? I saw my dr today bc I have a yeast infection and he told me to avoid sex while I'm irritated but got to thinking and wish I'd asked him about that too.