Not conceiving is God's way of punishing me for past sins

To make a long story short 7 years ago I stopped being part of the religious organization I was apart of because of the way I was treated. Seven years prior, to me leaving, I had sex out of marriage, which is considered the ultimate sin. The person I had sex with was someone outside of my religion, another sin. I was 19 and I was kicked out by the elders of the religious organization. The only good thing that came out of that was my daughter. So I continued to work on my relationship with God and after a year, I was accepted back in the church, by the elders. However, those members in the church still treated me like an outsider. I had no friends and I was just lonely. So I eventually met someone who was once again an outsider of my religion. I decided to leave the church, this time on my own terms. I continued to pursue the relationship I had with the man I met and together we had a son together. My mother was outraged. Eventually we got married and he considers my daughter his, as well. So fast forward to now, I am 35 trying to have my third, with my husband and just struggling to conceive. I can't help but remember my mother telling me that because I stopped serving God, nothing works out for me. I feel that she is right because the doctor's have not found a medical reason for me not being able to conceive. Now I am not a bad person I just don't belong to any religious organization. I just feel that now that I am married and trying to get pregnant I won't as part of my punishment for the things I did in the past. I feel hopeless....and defeated. 😢