My boyfriend acts like I'm always mad at him

Okay, so I've been dating this guy for a year. Obviously we have arguments. But he just always says things like "You're always irritated with me." And then tonight, he said he couldn't come over so I said "okay, then I'll just see you Monday or something" because we both have things to do all weekend. And he just started being so mean to me and acting like I was mad at him. And now I just feel so sad. I do get irritated at him sometimes, but usually when I do it's for the same reasons- he talks in detail about girls he's had sex with before, we argue about him being friends with his ex, stuff like that. So it's not like when I get irritated I get mad for no reason. I just feel like I'm a bad girlfriend and that I should try being irritated with him less, but at the same time I feel like I have valid reasons to be upset with him and he's guilt tripping me so I won't be mad at him. I just feel so sad and like a horrible person now, but I don't want to hide my feelings- especially if he keeps doing things he knows make me uncomfortable.  And tonight I don't think it's awful for me to have been a little bummed out when he said he couldn't hang out because he already had plans- I haven't seen him since monday and since he only lives 10 minutes away it's a little upsetting that we have to go a week without seeing each other. But it's not like I was angry with him. I'm just really sad