Husband gets mad for visiting family?

Welp. Since I got together with him my family was wanting to come visit my place and what not to see how everything was blah blah to him now he saw that as “man there here everyday should I bring out the cot?” I was like wow really. Meanwhile his family are the type that no one else matters but them. No one else has family like cousins or birthdays but them. Now I’m close to my mother and I like to see her and we go eat shop etc. his brother and sis live 10 min from there mother and father and are literally there everyday and on weekends. Today he tells me “now your going to be going to your mothers everyday right?” I said so what it’s not like your going with me what do you have to worry about. He then says “I don’t want my son over there all the time” I say well what was your excuse before I had him or before we got married??! He didn’t say nothing. Idk man in my heart I want to stay with him and for the sake of my son. I know what coming from a divorced family does and it’s not the life I want for him but man idk what to do anymore. Whenever it’s his family anything goes but with mine I can’t do nothing. Now I believe it’ll be harder with a house, cars, bank and a child to get a divorce. I know it could be worse but it only seems like it’s getting there pretty fast too. God help me.