It's complicated

Two and a half years ago I met a man almost 9 years older than me and I fell in love. I thought I would marry him one day. So we had sex. He had a ridiculously high sex drive and I was a virgin before him. I was 18. He said we didn't have to and that he respected my wishes but I was so eager to please him. I was never enough for him, he was never satisfied. We went on like that for a while and one day I had slept at his house (which was common for me) and he mentioned offhandedly that he had had sex with me while I was asleep. (I sleep very deeply) I was in shock. I asked him a lot about it and he thought nothing was wrong. He told me all about how I felt and how relaxed I was and that he knew I was unconscious the whole time. He told me he had done it before. I told him what he had done was wrong and that I was unable to give consent when I was unconscious and we talked about it a little while. After that I was paranoid and I still am. We stayed together and later on in the relationship he obliviously told me how hard it was to control himself when he had me asleep in his bed. We have been in an open relationship on and off. I have had panic attacks when people try to finger me. I've had several people ask if I've been raped or abused because of how anxious sexual contact has made me. I am currently on a break from him and trying to figure out how to break up with him after 2 1/2 years of being in a relationship with him and falling in love with him. I understand that this isn't healthy but it's so hard to leave him even after everything. Any advice?