Bf thinks I cheated.

My boyfriend thinks I cheated on him. He has bad trust issues and I know he does too so I try to understand. But tonight we were joking around about me being horny and I had plans to go to a friends for dinner, which her older brother and I had sex a few times and my boyfriend knows about it because it was before we got together. I told him how horny I was and to come over and he said "have Kevin" I'm assuming he was joking but I took it serious. I just replied with really and he said "u want to don't u" and I said no if I wanted to I would've when he asked me but I'm with u so I told him no. If I said no he would've said I do and either way it would've resulted into a fight. He now thinks I didn't say no, that I cheated on him. I told him over and over how I feel about him and I can't cheat on someone I love. But he doesn't believe me. Now I doubt we will even speak to each other tomorrow unless I say something first. I told him I can't be with someone who doesn't believe me about something this serious. And he just replied with idk. I said he needed to talk to me about it and work it out rather then reply with "yea" and "idk" and all he said back was I don't feel like it. It was something so stupid just a little joking around and now I'm not even 100% positive that were together right now. I'm just ranting because at the moment we aren't talking because he went to bed.