Confused

I just don't know what to do. I'm 32yo. I've been irregular all my life. Recently I took Provera to induce my period and then took Clomid. I told my husband before starting the meds that once I'm off my period we should have sex. He said ok. I've been off my period for about two weeks now and no sex. I feel like I took the meds for no reason. A couple of days ago I brought it up to him and he had the nerve to tell me that I am the one with the problem that we aren't conceiving since the doc gave me the pills. How am I the problem when I'm trying and he doesn't even touch me? I'm just so confused. Am I wasting my time with this man?