Why do I feel this way?

Meganne
So, growing up I was in a religious family of Southern Baptists. My parents never purposely kept me sheltered, but I guess the influence of the rest of my family made me think that sex was wrong unless you're married. My first boyfriend really pushed the topic of sex to the point that after a year of dating, he yelled at me until I said yes. He had told me we were going to get married one day anyway so it didn't hurt my religious values. He dumped me two weeks later. 
It's been 4 years since then and I've finally found the man I'm going to marry, but those horrible thoughts still haunt me. I'm not even religious anymore, but part of me still feels like I'm doing something wrong and part of me is still scared to have sex at all. I love my boyfriend and know I want to be having sex with him and it's wonderful when we do, but why do I still feel so damaged and broken? Can anyone help me understand what's wrong with me?