Was I wrong for assuming or was my ex overreacting?

I work with my ex-boyfriend. Long story short he broke up with me a month ago. He contacted me a couple days before and apologized for what he's done to me. He wanted to spend time with me but he wanted to be good again as friends and it could still turn into something more but he didnt know yet. So I gave it a shot to see and it went well when we saw each other one day for 2 hrs. Second day we had a good time but he never kissed me but we had sex. Made small minor motions to show him I still wanted him by rubbing his back n stuff but he didn't do anything. Ik he cares for me cuz the end of the day he hugged me tightly for a while before going. Today at work we made small talk and hugged me before going home. He didnt message me after work yesterday but today he did. He wished me a good day cuz I had the day off today. Then invited me to his apartment tonight or early morning when his roommate was gone to work. I told him I would see him tomorrow and that we wouldn't have sex this time. He lost it thinking I would think wrongly of him. Thinking all he wanted was sex when reality he wanted to just spend time with me. Honestly I dont know if I fucked it up or he just turned it around on me for thinking that.