Can the smell of marijuana harm you?

I am not a smoker myself but I thought this group would be best for my question. Me and SO live with his parents and brother. We have a 18month old and 3 day old. I think his brother smokes marijuana in his bedroom. He does have the door shut and a window open with a towel under the door (so I am pretty sure he is) but when he opens the door i can smell it. My question is can the smell harm the babies at all? Sorry if its a silly question. I don't know much about it.