Infertility Sin?

I want a honest opinion nothing sugar coated.

I’m married & we both want children. After trying for 3 years I’ve been told by many people to give up then it will happen. I’ve tried so hard to give up but it only caused me to be depressed because I couldn’t get the thought & desire to leave me. I know in the Bible it says & “God remembered her.” I think he hears me and has answered many of my prayers but do you think he may have forgot about me or that I may be bothering him? I’m starting to feel like I should stop praying like I’m becoming annoying. I know my husband is getting a little weary. He is the oldest of 4, the only male and his mother has no grandchildren. I feel like it’s putting a strain on my relationship with others and they all treat me like it’s me. I feel like it’s something I’m missing. I’ve asked God for so much already and I’ve begged but now I’m stuck. My father was killed when I was 4 and my mother has always had health issues with dialysis so I’ve learned to depend of God at a young age. I know he is a jealous God and my past sins and me begging for a family may have upset him. I just don’t know what to think anymore. We use to be so close. I feel like this infertility situation has made me a little upset and our relationship isn’t as good as it used to be or could be.

I just don’t know what else to do. I really don’t know any women of God to talk to who may understand what I’m going through. I don’t want this to drive me crazy 😔 but I don’t know how how I should be feeling.