second hand smoke

My so refused for me and my one month old to drive 2 and a half hours to my dying grandpas house because he use to smoke cigarettes in his home and he doesn't want our child breathing in the second hand smoke. But yet his mother smokes weed multiple times through out her day in the house and he always wants our child to go over there. Does anyone see an issues with this? (Obviously I don't want my child breathing in second hand smoke I was breast feeding and didn't have a pump at the time. I quit smoking for my child but he made me miss saying my goodbyes to my grandpa because of that reason)