Sex during IVF?

My husband and I started IVF treatment in July. We had an egg retrieval earlier this month but we have to do another due to poor outcome. Since early July, we've only had sex a couple of times, because I was so uncomfortable from the stims and retrieval, then I got a period, and then we had a very short window until I started Lupron for the new cycle. Now I'm afraid to have sex while on the Lupron before we start stims. The doctor never explicitly told us no sex, but I'm not sure what happens if we have sex while on stims...what if I ovulate before the retrieval somehow and we get pregnant and then the retrieval hurts the baby? We don't use condoms, even before we were TTC (I had been on the pill, and we had difficulty with condoms for a few reasons)...I'm feeling the lack of intimacy though and it's kind of making the fact that we can't have a baby naturally even more depressing. We are going on vacation in October during a break from treatment so at least we'll have some time together before our transfer.

For those who have gone through IVF, did you have sex? Or did you just muddle through the abstinent period and look forward to when you had a break between stims and FET?