Upset with husband

So I have PCOS and my doctor has been putting me off for a few months saying I need to lose weight. I wasn't ovulating before which is why we haven't been able to get pregnant. I have lost 14 pounds and I am finally starting to ovulate. This is my first month ovulating so you can kinda understand my excitement. Anyway me and my husband are on separate shifts so it's a little hard for us to get time together but we do our best. Last night should have been the night we spent together but he then tells me he don't want to cause I only want to for one reason. I have PCOS everything has to be tracked? Am I in the wrong here? He did end up apologizing this morning but by then it was too late. What should I do?