Issues with SOs family

Brittany • Mommy to a wonderful 6yr old boy, 2yr old girl and 1yr old girl.
My fiances dad smokes in their house. I don't allow my children around cigarettes or vape pens nothing. I don't even want them going over there if he's not smokeing cuz it still smells and there's tar and nasty all over everything still. His dad knows I don't allow this. I wouldn't tell him he can't smoke In his house but I can keep my children from going over there. It has become an issue of I keep my children from seeing their grandparents. Which is not the case. We invite them to our house every weekend and they want us to come to them. My SO's mom also has shingles right now. So I don't want my kids around that cuz it can cause them to get chicken pox. I have a 5mth old so I'm real careful about that. I just feel that I shouldn't have to put my children's health at risk to make others happy. Am I wrong for this.