Saying 'No' in a relationship

I love my boyfriend a lot and I know he loves me. When we joke around sometimes he can get kinda rough and he grabs for my butt or my vagina and I get uncomfortable and anxious with it. I tell him no and stop but he doesn't stop until he's ready to stop or I freak out a little. When I try and tell him after that I said stop and no and he didn't, I try to express how that makes me feel and I don't like it, he gets mad and turns things around onto me. I don't really know what more I can do. I get frustrated and mad myself and I feel as though he doesn't care about my feelings of what I say. Is there anything more that I should be doing? Or what I can do?