Sex while pregnant?

I'm currently 4 weeks 3 days (I know super early) but I'm so scared to have sex! I feel like something is going to go wrong or it will cause a miscarriage or something! My SO wants to & I feel bad because the only reason I won't is because I'm scared lol. Someone tell me I'm crazy & it's perfectly okay to do it?!?!