Me and my man had a fight

So we’re just chillin at lunch and everything and he’s got his hand on my thigh which I’m totally fine with but he start to get to handzy and it was make me uncomfortable and I told him to stop and he just kept trying and kept telling him to stop and eventually he got really mad and threw my hand cuz I was holding his hand trying to get him to stop and couple minutes later he tired again and I told him to stop again and he got mad again and just layed his head down on the table then we left lunch and I didn’t look at him or talk to him until I cooled off before I said something I didn’t mean then I saw him a bit later and he asked if I was mad and I said yea and he asked why but I wouldn’t tell him I wanted him to figure it out himself then he figured it out and I told him why I got mad and that it made me very uncomfortable cuz of things in my past that I’m pretty positive he knows about. And my friend was sitting right across from us and I was listening to her talking and not paying attention to him after like the 3rd time I told him to stop. Then he said after I told him why I was mad he said he felt like I was wanting to listen to her and not pay attention to him. Idk if I was wrong here or what cux I feel really bad and I texted him and now he seems really mad at me

I need help I really am trying In this relationship and my parents like him and that means a lot and idk what to do