I NEED ADVICE...and I'm so ashamed.

I've been "dating" a man for over 3 years. He recently spazzed on me because I told him that the fact he complained about me giving him blue balls when I hardly do after not speaking to me for over a week was a turn off. He said we're not in a serious relationship which I understand. But then he went on to say he doesn't like that I talk or spend time with my male friends. I hardly do but being that I'm not in a serious relationship I am free to do whatever. I just need some justification and tips to kick his ass to the curb.