Dealing with my Husband BabyMomma

Any one else gotta deal with a BM who allows the child around their dad SOMETIMES...One min the child is allowed around you and your husband then the next min he isn't without any explanation or whenever she gets mad...Its soooo draining...I have a little one too but I wonder if as he grows he will ask why can't he see his brother one minute and the next he can....My baby is young still but he only been around his brother once... It just feels like a broken family so I'm starting to only focus on my hubby and my child...Since I never know when his brother will be around....So annoyinggg