Can anyone help me out here?

I had my baby May 26th. My family in law has asked us to come over every weekend since baby arrived. At first it was okay because they’re excited about their first grand baby and I understand that. I can’t keep doing it. My hubby is already back at work and I only get weekends with the three of us. I tried to explain to him that I want us to have some time together just us and he’s like you can tell them that then. Like no dude, you manage your family I manage mine. I don’t want to be the bad guy. I feel like my husband just doesn’t understand. I want to sit home and relax and keep to my schedule with him. How can I help him understand? Or should I just be the bad guy?