Pot fumes while pregnant?

I don't smoke pot, nor does anyone living with me. However, I live in a duplex type house and our downstairs neighbors and upstairs smoke. I haven't been noticing it lately but tonight the whole apartment smells from them smoking it. I know this is a topic that's controversial but I wanted to make sure there's no harm to the baby? I'm moving early December and starting to feel like it can't come soon enough.