Threesome when with me?

I'm losing my bloody mind! When me and my OH first got together we had talked about our fantasies etc he said his was a threesome but he'd never do it because he was too self conscious and he thought too much of me and that was the end of that. 6 months into our relationship and some girl messaged him about coming to a party and he messaged "what do I get? A threesome with you both?" He's quite a jokey person and put it off as he was being sarcastic and didn't go. We moved intogether after 14 months and we had friends around playing a few drinking games and one of the questions on the game was to drink to figures if you'd had a threesome and of course my OH drank, he never did on this question.im losing my mind that he done it when with me but he said it was before he even knew about me at a party he walked into 2 girls and joined (yea right) he kept suggesting that my friend when I lived in a shared house stayed beside us until her bed arrived or that we all went up stairs to watch a movie in bed. I don't know what to do