Is my relationship toxic?

So I’ve been with this guy since we were both 16 years old. In the beginning, he was so sweet, always listened to me when I was going through bad days (I have a lot of mental issues), and was understanding with everything. Now four years later all we mainly do is argue, he doesn’t listen to anything I say anymore, calls me annoying at times, and when I recommend dates that I think would be fun he says, and I quote, “that’s doing too much. Let’s just go to my house”. He always expects sex and when I say I’m not in the mood or I just don’t feel like it, he gets upset and kind of throws a fit. I’m seeing red flags that I never saw in the beginning and i don’t know what to do. I don’t want to leave the relationship because we’ve worked so hard on this relationship but at the same time I feel like it’s not helping me mentally. I’m going to therapy and when I told him, he said “you just need medicine, at least that’s what I think” but didn’t ask me for my opinion. Then there are times where it feels like he’s trying to control me, control what I do, and control who I hang out with. When I told him how I felt, it led to an argument and that’s what usually leads to our arguments. I express my feelings, he doesn’t like the way I explained it, and he gets upset leading to an argument. I can slowly feel my mind losing feelings for him but my heart and soul can’t give up. Any tips?

Edit: he’s mad because my friend is celebrating her birthday this weekend by going out to eat and going to club. I told him I was thinking about not going but decided to go and now he’s mad. I’m honestly over at this point because when he’s mad, I always say sorry and tell him I’m not gonna do whatever it was that mad him mad but I shouldn’t have to keep stopping my life because he doesn’t like it. I’ve been doing that the entire 4 years we’ve been together and I no longer have friends to hang out with anymore because they tell me that they don’t want to cause problems with me and him.