When is the right time? & am I dumb ?

So I’ve been dealing with this man for 2 years he has 2 kids.. when him & I first got together he told me about them they stay with him off and on.... months went by I never saw them but I know how it is with kids so I didn’t rush to meet them and he told me his oldest son was super sensitive about his dad and his mom not being together so I never rushed to meet him. The guy always came to my house or i came home when his kids came over... little did I know the mother of his children... super BITTER! I honestly thought he was over exaggerating the situation until I saw it first hand.. so I understood why he did certain things.. but now it’s 2 years and it’s the same thing I can’t be around his kids because his baby mama claims she’s going to “beat me up”.. I feel as if he is making excuses about a lot of things because the situation may be deeper then what he’s telling me... I’m not with him to cause any drama or play step mom to his kids... I don’t know if I’m overreacting at all I just need a opinion.. I’m really just tired and stressed out.. when his kids come over I stay in the room or he takes me home and I can’t see him for days... I’m just sick of it