Sex after IVF pregnancy!

Hi ladies, my husband and I conceived after two rounds of IVF due to a male infertility factor. I am 13 weeks along and was told that because we conceived through IVF the pregnancy is considered 'high risk' even though the baby and my uterus are healthy and normal. We've talked to two different doctors about intercourse, and neither want to give a direct answer if intercourse is allowed. I am going crazy! Do any of you have restrictions?