Open Marriage

My husband and I are very open sexually. We've had threesomes before and he encourages me to keep exploring. He's gone for work and he said I should invite another women over. Last night I asked if it had to be a women and he said either way just have fun. So another man came over last night. It was terrible but that's beside the point. My husband messages me afterwards and says we won't TTC this month because he wants to make sure it's his (valid but I used a condom) and how I may have an STD now and need to be healthy first. We've been TTC for 2 months and he's been encouraging this sexual curiosity but now that I've been curious he's sprung these thoughts on me. I'm just so frustrated that he waited to voice these concerns until afterwards!!! I booked a flight a couple weeks ago to go see him next weekend during my fertile week. Now I'm going down to what? Not have sex with my husband and be pissed at him for not ttc? He has ejaculation dysfunction so it's hard to conceive in the first place. I'm just so upset with the timing of all this.