confusion

my male partner has told me he's had a smoking past, slept around & had a 2 year relationship with a girl who had a baby with someone else. now we've been in a committed relationship for 7 months but my self confidence isn't strong & I'm trying to get rid of his past bc it has effected me. he lost his virginity at 18 & I lost mine when I was ready at 22. yes we have different past lives & he says it's made him into the person he is now; successful engineer & great lover but am I crazy for thinking about his past bc it wasn't like mine at all?