I need Advice Please!!!!!!

So I have shared custody with my first son who is 4. His father is married and has other kids. My son always cries when it’s time to go to his fathers house. I ask him why an he tells me he just wants to stay with me. My son also tells me stuff like he eats cereal for dinner... Now my son is 4 but he can’t be lying to me. Also one time I was in a bad relationship where I was extremely unhappy and my son never wanted to be at my house, he just wanted to be at his dads house. So that makes me think something is going on at his dads house that makes him feel uncomfortable... now if I bring it up to his father he lets the wife get involved and I have already went to jail for her (aggravated battery) not proud of that. Also I’m pregnant 10 weeks and 3 days and I really don’t need the stress.

Ladies please help!!!