Advice please

So me and my boyfriend have been together for a few months. We are currently living together, he is gone for a traveling job 3 weeks out of the month. When he is home we have his 2 kids and my son for a full week. I've recently noticed his children becoming more aggressive with my son. His son is 3 and his dauguter is 1 1/2. My son in 10 months old. They smack him, his oldest has kicked my son and thrown rocks at him. We've talked about how we are okay with disciplining each others kids. However I am the only one to ever point out how his kids are towards my son. He sees it and will adress it if I ask him to because i dont feel right being the disciplinarian for his children. He is divorced from the mother of his children and she is extremely rude towards me and is constantly calling into question my pregnancy. I'm 6 months pregnant, my boyfriend was aware before we started dating. She is still wearing her wedding ring he gave her even though she is living with her girlfriend. With this pregnancy I've become alot more self cautious especially with her behavior and how he refuses to confront her or tell her to stop because he dont want to cause conflict. He talks about wanting another child one day but we have 4 together (after my dauguter is born). He is good to my son and uses the same nickname for him that he has for his kids and never makes a difference between them. However even with all of that I am still worried he will leave me for his ex wife since she clearly wants him back and he won't acknowledge the rude and degrading text messages she has sent me concerning my son my daughter and myself. I am so confused right now. I want to leave because of all the red flags but he also makes me happy outside of the problems and we get along great. I'm second quessing if its pregnancy hormones or not.