my husband wants to keep keys to his babymothers house

I am 6 months pregnant and my husband has two kids with his previous girl. They use to live together for 10 years and we live together for and have a baby on the way. I took away the keys to his babymothers house and he is highly upset because he says his kids live there and he helps her with rent because his kids need it. I've been crying for hours I'm so annoyed and pist. Makes me want to call it quits .. please help ... :(