Relationship Advice

I just need an opinion on something. I told my boyfriend this morning I didn’t want to do anything sexual because I was tired and he was about to go home soon. He respects when I say no, but he just seemed irritated that I didn’t want to do anything. He went home but he still seemed irritated I tried talking to him about it and he kept telling me not to worry about it or that it was okay but the way he says it says otherwise. He finally told me that he was upset that I said no. I have been r*ped before and I’ve told him. I don’t know if he forgot or anything.. but I don’t even know what to say to him