My boyfriend is friends with his ex, should I be worried.

So, bit of a back story. My boyfriend has done some things in the past which has taken away a lot of trust from our relationship.

He is still friends with a girl he once had a sexual relationship with, not an ex as such just sex and dates. He goes to see her and her new boyfriend and doesn’t get why I’m not comfortable with it.

I don’t want to be a control freak, or lose him but I just feel there is a way to compromise on this given how uncomfortable I am with it.

When we first got together he would talk about her A LOT. I just feel our relationship would work better if he segregated his past from his future.

Please help. Any suggestions welcome.