Question about sex after 2nd trimester miscarriage?

So, my doctor told me to avoid sex for about 6 weeks after I experienced a 2nd trimester miscarriage. (After delivery, I only bled for 6 days or so.) Then, at about 2 weeks after the delivery, my husband and I had sex 😣. Should I be worried? Any ladies here have sex before the recommended wait time?