Finally a positive OPK and my husband changes his mind

I have PCOS and have only ovulated 3 times this YEAR.  One ended in an ectopic pregnancy in April.  My husband kept saying we would try again.  I finally get a positive opk and then he says he doesn't know if he wants another baby and we don't have sex the night of the positive.  How is this fair?  It's driving me crazy.