Husband thinks I’m lying

My husband and I have been married for 8 years. Recently we where having a conversation about sex and it ended up with him telling me I used to live with my ex of four years. Context I was with someone for four years before him. It came to me as a shocker and my response was well I never lived with him. And he said stop lying and I said well I didn’t. He told me my brother had mentioned it to him, which I am really pissed at him right now. First of all because it’s not true. Never did I live with him. Never did I sleep over. We did go camping at times, but it was not something that we did often. Regardless of it my husband says I’m lying and I am so pissed and needs to get it out some way…..