Am I the only one in this boat?

My husband keeps shutting me down when it comes to trying to conceive. My gyno told me I should initiate sex seeing as my husband doesn't have as high of sex drive as I do and I can't tell him its my fertile days as he seems to think of it as a chore. Well each time I try to get things started he will give me a quick peck and go to sleep and if I persist he will get annoyed with whatever I do. I have spoken to him about this and each month he says oh ok I didn't realise. He is getting annoyed that we haven't conceived and I just feel like telling him if he would help me out we might get somewhere. I feel like I am the only one putting effort into this and it has now become a chore that I am really getting tired of.