Am I in the wrong

Every week my husband wants to go out with his friends to use his RC car. They all have put money into them and like using them. Today I told him that I don’t feel like going because last week he invited me and I was literally left alone with our baby. I was pretty much chasing our baby around while him and his friends did their thing. I told him he can go and me and the baby will go to the park that way we’re not just at home.

Well he got super mad and started cussing and saying that he wants us to go and why I have to make everything difficult. I just didn’t think it was cool to invite us and pretty much leave us there. I feel like everytime it has to do with his friends, if I don’t want him to leave so he can be with us or if I don’t wanna go, it becomes such a huge argument. Two weeks ago I had asked him to stay Bc I wanted to just hang out and he was pissed then last week he wanted to go out on Saturday with his friends to do the same thing and I told him Sunday was better since we had things to do and again he got super pissed. Idk what to do anymore. I feel like he treasures his time more with his friends than with us. Considering he does work with the same friends and is gone 5 days sometimes 6 days a week.