Sex before getting married?

So my hubby is the 4th guy I've slept with, and I'm his first. I don't think about or care about the other 3 guys because I know that at least 2 of the 3 were not out of love, and the guy just wanted to get some. The exception being my first, but it was only the one time and we were young and in love and a little scared of getting caught. I'm his first. He told me he was waiting until he found someone he'd actually want to be with not just do it with. I was 16-17, and totally just trying to find my place in the world too early. I feel bad, but it bothers him occasionally when we are hanging out with his friends and they bring up how many people everyone has slept with. Even without me pitching in I can see it makes him uncomfortable, and when we get home he always asks me if I ever think of the other 3. I don't like feeling guilty, especially since we've been married over 2 years now and are ttc.

Sorry for the long story, but any advice ladies? Please don't be rude or judgemental, I just need to know if I'm going to have an issue on my hands or if I'm just paranoid..