How often do you talk about your exes with your current SO

Just wondering! I've slept with quite a few men in the past, but only had one long-term relationship in the past, before I met my husband. My husband has had a LOT of previous relationships. It annoys me sometimes when he'll mention someone that he dated in the past. We'll be talking about something like possible jobs, and then he'll be like: "I dated a girl one time who had a food cart..", and then he'll talk more about her. I kind of let it slide, but when I talk about my past relationship of nearly ten years, he gets a pained look on his face. I am usually just making a brief reference, like "I travelled there with him one time". I think we should have a discussion about just never bringing up exes (except for his two ex-girlfriends that he has kids with).