Did my abortion ruin my change of conceiving again?

When I was a teenager, I was careless and got pregnant by this guy I have been engaging with for almost 2 years. He always made me feel like he actually liked me, until I told him I was pregnant. After that, he treated me like I was a piece of dirt. So of course I did not want to have his baby. I went and had an abortion. I am now 23 and my fiance and I have been trying to conceive for almost 2 years. I feel like having the abortion ruined my chance of ever having a baby. Am I right or am I just thinking too hard about it? My doctor said it is nothing to worry about but I still think about it everytime I get my cycle.