Tired of hearing about the ex

My partner has two kids with his ex who he's still legally married to and shares ownership of the home she lives in with their kids I on the other hand have a son and as of over a year have no contact with his father because he chose he didn't want to be a part of his sons life. Now here's my struggle they were together for about 15 years before the final split and at times when we talk or are at his mothers home conversation will come up which has to do with the past and he uses the word we referring to his ex and him for example  if the talk is referring to ice cream he would say when we went to Texas there was an ice cream stand and it bothers me because I feel that for my consideration he should adjust it to when I went to Texas and I know that it was his past and they did it together but it bothers me that she's made present in a conversation that he can adjust for my comfort when we're in front of others who have a relationship with her as well ..it's like she's always there without being there. Am I wrong for asking him to watch the way he says things ... His mother has a tendency of speaking of his ex as well and don't get me wrong his mom is a great lady and she doesn't do it to be mean or purposely make me uncomfortable ...since it is the mother of her grandchildren I get it but I feel disrespected and think that she shouldn't be mentioned constantly sometimes they'll even ask me about her I think I made a mistake by being the cool open understanding new girlfriend and now I don't know how to approach it the reality is I just want to yell for everyone to shut the fuck when this happens but it's wrong to do something like that I don't know how to mention it to his mom and him