Am I wrong?

My ex boyfriend and I were together for almost 2 years when we broke up. We were living together at the time so when we broke up I moved in with my cousin. During the time we’ve been broken up, I had started seeing a guy and things moved quickly with him in my bedroom but I ended things with him because I missed my ex dearly. Well now me and my ex have decided to try and work things out. But we will continue to live separately until we get engaged or married. So because of the separate living arrangement I asked if he would come stay with me to which he responded he wouldn’t come stay with me until I got a new mattress because he just can’t bear to sleep in the bed where I had sex with another man. I feel like he’s being a bit ridiculous considering before we got together he had been married for 4 years and I never said anything about sleep in the bed where they had had sex. Am I crazy? (He is 33 and I am 23).