Who's in the wrong?

Basically I live with my SO and promised him he could sleep with me whenever he wanted to (I'm too tired sometimes) and have never stopped him doing so, however tonight I told him no several times, he continued to carry on. I then told him it was hurting me and he still carried on. It was only when I was crying that he stopped. My ex basically used to force me into sleeping with him and my SO knows that. He thinks he's not done anything wrong because I promised him he could do it whenever he wanted which is true however common sense would of told me of I said no then to not carry on. Who's in the wrong please girls, me or him? I'm doing this to settle the argument.