Going on vacation with the in laws

My husband wants us to go on vacation to DR with his parents. He said it will be good because they can help with watching the Kids (two kids, age 7 and 18 months). I don't know how to feel about it but I'm leaning towards ...hell no?

The point of vacation is to get a break from everyone. His in laws are so intrusive already as it is (they live 2 min away from us, walking distance) that thinking about having them during a relaxing vacation makes me nauseous.

Am I crazy? He seemed pretty offended when I told him I didn't want them to go with us.