Regret abortion?

Hello, I had a very early abortion at 4 weeks (took the 2 pills) about 4 years ago when I had just gotten married, was changing jobs, under tons of stress, and moving to a new city. I got pregnant on birth control pills and was drinking heavily due to stress (hence the move out of a busy city and new job). At the time, I felt like I made the right decision and my husband agrees. Flash forward to now - We have now been in an ideal place physically, financially and emotionally to get pregnant and have been TTC for about 3 months with no luck. I go through waves of depression thinking that I'm being somehow punished for my decision 4 years ago and missed out on my baby. I didn't think it would be this hard to conceive when I got pregnant on birth control. Has anyone else gone through something like this? How did you deal with the emotional side/feeling like you're being punished? No hateful pro-life comments here please...