Should I stop trying?

So at the beginning of the year my husband and I agreed to try for a baby. Went to the doctor and started clomid. The doctor advised it would take about 6 cycles. Well I just finished my 3rd cycle. This cycle my husband has seemed to be more distant. He even told me if this cycle doesn't work he don't want to try anymore. We had sex once during my ovulation. He's been working crazy hours due to swing shift, sometimes we foo days without seeing each other. If I'm sleeping when he comes home he tries to make sure he doesn't wake me. If he's home before me he's asleep before I get out the shower. I'm so frustrated at this point and I feel he is intentionally screwing up this cycle. We've been married 6 years and our sex life had never been so distant. I don't know if I should stop trying or not at this point.