Would I be in the wrong to leave?

My boyfriend (who I’m also pregnant by-7 months pregnant) let his brother (who is older) move in the apartment with us because he got kicked out and is facing homelessness. His brother is a heroin addict, last night around 11pm my boyfriend went through his brothers bag and found a sandwich baggie tied in a knot but in that baggie was a white powdery substance, didn’t look like cocaine as cocaine literally looks like crystals, not powder. So we came to the conclusion that it’s Fentanyl and/or “China White” heroin.

2 hours go by and we’re on the way to Jack in the box to get some food, I asked my boyfriend what he plans on doing about the situation (this isn’t the first time we found paraphernalia in the house- I found it the first time..) He doesn’t say much and he asks me what I would do, I bluntly told him how I felt, how I’m not willing to be around anything hard like that- Weed is one thing. But crack, Meth, coke, etc.. I’m not okay with. God knows how many police officers died or OD from a traffic stop behind these things..

However In the midst of the conversation, he tells me it has nothing to do with me, how his brother isn’t my family etc.. I then remind him how we have a baby due soon, I remind him that I’m also in the house and around these things.

At this point he’s pissed off and I tell him to just go back home and I’ll go get myself some food in my own car etc.. We get to the apartments but on the side of the street where we park our cars (almost there) and he tells me to “Get the fuck out” and tells me “I’m lucky that he doesn’t slap me in the face or beat my ass behind his family”.

He still didn’t put his foot down, his brother doesn’t do shit with himself, he doesn’t apply for jobs or anything and I’m just fed up truth be told..

Would I be wrong if I packed my stuff and totally left?