Sex while Preggers

I'm 6 weeks pregnant today with my rainbow baby. I'm so scared I'm going to miscarry so I'm afraid to have sex. My doctor hasn't told me I can't have it and I feel like I'm depriving my poor husband but I'm just so scared I'm going to miscarry. Not meant to be TMI but he isn't "small" and I'm scared that his manhood will mess something up but again I feel like I'm depriving him and I HATE that :( what should I do?