Would you stay with a guy who has a son from a previous relationship if he goes over to his exes house to see his son and refused to go to court?

Let me explain a bit. My boyfriend and I have been together for a while. I also have a son from a previous relationship so I understand having to stay in contact with the other parent. However in my case My sons father sees him on his own time, he doesn’t come over to see him, and there’s really no drama or hard feelings. My boyfriends ex/baby mama is a different story. When she found out that we were together she lost it, threatened to take his son to another state, etc. when she found out I was pregnant she said he could no longer have his son on his own. She blamed it on Covid said that he couldn’t come over because it wasn’t safe when she was just fine with him coming before she found out I was pregnant. Also shes taken him to theme parks and even wants to take him to Mexico so the whole Covid excuse is bs. She said she wasn’t keeping his son from him because hes still more than welcome to go over there to see him.

So he goes over to see his son right now once or twice a week. This is been going on for about two months and he refuses to go to court about it. He pays 500 a month in child support, his son is on his insurance, he’s always been in his life and he checks on his son so I don’t understand why he will not go to court. I don’t know why he is OK with having to go to his exes house to see his son when it shouldn’t be that way. She is only saying that he has to go over there because she is mad about the situation. I asked him why he does not want to go to court and he says it’s a waste of time and money.

He tells me it’s not a big deal, because he goes over there to see his son not her. But that’s not even my point. He’s letting her have her way, letting her control him because they have a kid together, when he could do something about it instead. I’m honestly so tired of it and ready to leave. Am I overreacting? Would you stay?