Having troubles letting go of his past..

I need some advice on how to not let his past get to me.. my boyfriend and I have a 6 month old together, we've been together for only 2 years and he is about 8 years older than me. I know when he was in high school and college that he was one hell of a play boy.. always at the bars and clubs always hookin up with chicks and he hasn't been in that scene for years so I know he's not out cheating but it just bothers me when people bring it up !! His sister says to me " thanks to my brother being such a player I get free drinks at the bar" or "your son is such a flirt like his daddy" " he's gunna be just like his dad, a ladies man" and it is starting to low key really bother me... I just don't know how to let it go or not take it too personally. All of his female friends are way older than me and I feel like they don't respect our relationship because I'm so much younger than him. A few have called him to say this girl or that girl was mad because he had a baby wth me not them and when I confront him he just laughs and says they were never together just girls that are jealous and I shouldn't be worried but fuck how can I not ?!