Asking for threesome?

My SO is really jealous. He doesnt trust any of my male friends but anyway I am independent and we always talk and work it out because it's not like I give rights to any of them. I know he is insecure and try to help him and he also helps me with my own insecurities. So he has this threesome fantasy, us and another man. This is also a fantasy of mine but I would never try that with my SO. If I was single and dating I would think about it -fantasies are far from reality-but now hell no! I have asked him would u like this with me in real he said "no I want u for me but it's weird how my insecurity messes with this fantasy". But almost everytime we are having sex he will ask "do u want another man with us? " such things. I know he asks only because he is horny but again I feel like if I say yeah he will get pissed. And also I don't want to say yes even for fun because when we were talking (out of bed) and I said I don't want to try this with him I added that if he persisted I would think about it cause he is my SO and he got pissed. And now every time he has to ask me the same thing while we're having sex??! What am I supposed to say