Relationship Issues

So my boyfriend keeps talking about my exes and asks me whether I’m still attracted to them or not. If I don’t answer he automatically says that I must be since I didn’t answer. I get so pissed off because I don’t understand why he is so focused on my past relationships. I mean he’s going through a divorce and has three kids for crying out loud but I never throw that in his face and make a big deal out of it. It’s gotten to the point of where I can barely talk to him. I’m running out of ways to handle these conversations and it’s gotten to the point of me on the verge of crying because it’s causing me so much stress. Any advice?