How often do you go to see your children's grandparents?

My husband and I are always going back and fourth on this. He works ALL the time, and he always wants to go to his parents on his day off...he says we have to let them see the kids. I say I think going once every other week would be fair, seeing as I dont go to my moms every week, either. He says if I don't want to go down there once a week, I should drop them off once a week at least. I don't want to do that either? I say that they can come to our house when my husband is home, (we don't speak the same language so if he isn't home we just sit there lol) and see the kids...but he says his dad won't drive his mom so it's our job to bring the kids to his mom. I just don't want to go over there and just sit every week. I'm willing to go every other weekend, but am I being selfish, or rude, etc?