smoking in laws- newborn.

My husband and I do not smoke and do not allow smoking in our home. My mother-in-law and father -in-law are chain smokers and they both smoke in their home. They cannot go a hour without a cigarette. Roughly its 4 an hour. Is it wrong that I don't want to spend much time there after the baby is born?  I don't go to visit now I'm pregnant if I do I stay no longer than 10 minutes. I don't want to cause a rift, but I also want to protect our new little one. I don't want my baby breathing it in. I know it's awful of me, but I don't want them to hold my baby for long. As third hand smoking is so dangerous.

I've talked about it to my husband and he's said 

"If I want to take my baby up their house I will"

He's not understanding at all. I feel like I'm alone on this and overreacting.

Help help help 😭😭😭😭😭