So annoyed

So I have panic attack disorder, agoraphobia, and anxiety. On top of all of this I'm an introvert and have been my whole life.. Well my issue is my bf doesn't understand these problems at all. I don't like being around a bunch of people or people I don't know. I don't like meeting new people bc it makes me uncomfortable and I've told my bf this so many times. He assumes I'm rude bc I don't just go up and start talking to his family (I've only met his grandmother, daughter, and father so far).. I told him if they talk to me I will respond but I'm not just going to start talking to them bc it makes me uncomfortable. I really just want him to understand bc it is really getting annoying that I have to repeat myself constantly about why I am the way I am.. I told him he is literally the only man I've ever dated who is so offended I don't talk to people or like being around them.. I also told him I'm tired of him assuming I'm just rude bc he doesn't understand how my mental illnesses work! I can't control how I feel and I wish he would just understand that!