Bipolar or just mean

My SO and I have been together for two years, married for one. I have been trying to deal with his bipolar disorder for about a year and a half. I am now 11 weeks pregnant and high risk. At first he seemed excited about the pregnancy since he is 40 with no kids. He is now acting as if he doesn't care at all. The doctor has expressed that the stress he is causing by calling me names, saying he is leaving, and withholding affection is putting more stress on the baby. I love him and wish I could help, but everything I do seems to be wrong. He is mean to his parents when he doesn't get what he wants, he is mean to me and my other two children and runs the house like a warden. I need advice, please.