Guilt over previous termination of pregnancy

Unsure if this topic is in the right place. I certainly don't want to offend or upset anyone. 
I'm 26. SO and I have been ttc for 7 months. When we first got together I was just finishing high school and got pregnant and terminated the pregnancy and I'm so scared that having that done all those years ago has somehow impacted our ability to conceive now. Other then a few family members, no one knows about it, not even my best friends. They say that when ttc you shouldn't look back on stuff like that and feel guilt but I can't help it...anyone else ever been through the same thing?
Had we gone through with the pregnancy we wouldn't have half of what we do now and it was the right decision at the time but just hard knowing our family could have been started 😔