Bear with me- this is lengthy!

My husband and I have been married for almost 2 years and together for 3 years. Since day 1 we have talked about buying our own home since to my displeasure, he lived in our current rented house with his ex (mother of his first two daughters). I am now almost 20 weeks with our first baby together and I’m becoming increasingly agitated at our living situation. This is a 2 bedroom house, a very small house I might add. There is a tiny section in the back of the house that used to be a porch that was closed in and is now a teeny-tiny room that leads to the bathroom, the only access to the bathroom I might add. He wants to move his kids out of the big, real bedroom and into the tiny room which is fine and dandy if I and all my guest didn’t have to walk through two prepubescent girls bedrooms to get to the bathroom.. their room is always a disaster area so you can see my issue with this, not to mention it has no door and is open to a main part of the house. I feel overrun, my space feels invaded and I’m becoming very frustrated and angry with my husband at his unwillingness to want to buy a house suitable for our child and our family. This house is not safe for an infant- we recently had to replace almost all of the floors because I fell through the floor not once, bur three times in three separate rooms, there’s also a room where you can see under the house through the floorboards, there’s no insulation, the windows have no locks or seals- this house is a ticking time bomb for an accident with a baby and for me to lose my sanity. I just don’t feel he’s taking anything very seriously and I just feel so beat down and depressed over it. I don’t know what to do anymore.