Should I have to choose between my spouse and parents?

UPDATE:

So I talked to my husband tonight and told him I was going and taking the baby. He is welcome to come with us, but that he doesn’t have to. This caused a HUGE argument and he basically said that I can go to my parents, but I can’t take his daughter. He threw every little thing my parents have ever done wrong in my face and gave every excuse why we shouldn’t go. He also threw a cup against the wall and stormed out of the room like a child. We argued some more after that and I told him that he needs to tolerate them and their flaws and let go of the hate or just end our marriage now. Then I said neither one of us are feeling heard right now and so I walked away. He then grabbed his car keys and left. He has never left before. I’m not sure what to do now or how to make this better, but I’m trying to stand my ground. 😭

It’s not like my parents are abusive, toxic people. They are rude at times and can be selfish, but they are family so I look past that. I have had conversations with them as well about things they do that bother us, so it’s not like I’m not trying. He thinks I see them through rose-colored glasses, but I think he villainizes everything they do.

Sorry this is long-

My husband and I have been married for 8 years, together for 11. He hates my family! They are loud, from New York, and they bicker a lot. Always a lot of drama. My parents used to argue and yell a lot when I was growing up, but they have really worked on this over the years. Now they just bicker and talk loudly, but to me, they are so much better.

My husband is southern and his parents are quiet and soft-spoken. It’s hard to believe, but they pretty much never argued in front of my husband ever. So because of this, my husband hates how loud and annoying my family can be.

The root of the issue is that my mother kept a secret friendship with my ex-boyfriend around the time that I was getting married. She talked to him on the phone all the time and gave him rides places because she “felt bad.” She also tried to persuade me to go back to my ex once. This was really messed up and when we found out, we didn’t talk to her for months. She went to counseling and we eventually forgave her... well at least, I did. This was 7-8 years ago and my husband still won’t let it go.

My parents live 3 hours away. Anytime they want to come visit or invite us over, my husband and I argue over it. Pretty much all of our arguments have to do with my parents. He would prefer that we only see them 2x a year. We have a toddler and my parents want to see her as much as possible. They were coming every few weeks before the Coronavirus, but I finally told them that it was too much and to space out the visits. This has caused a lot of tension because anytime they ask to come or invite us over, my husband doesn’t want to and we argue about it. Now my parents feel like we have pushed them away.

My grandma (who lives with my parents) wants us to come visit soon for her 89th birthday. My husband put his foot down and said my family is trying to manipulate me into visiting. He claims he’s not trying to keep me from them, but that they should only get visits twice a year. I genuinely want to go and want them to see my daughter, whom they haven’t seen in a while. I feel like he is making me choose between him and them andI hate it!

*Also, I am concerned about seeing them right now because of COVID, but we have been social distancing and so has my family.

EDIT: his parents moved here when the baby was born. They watched her while I was at work and now they still see us multiple times a week.