Am I wrong?

All right girls I need your opinion on something because if I’m in the wrong then I’m in the wrong and I can admit that and accept it. I am with this new guy he has kids I have kids we were both with our exes for the same amount of time. He was very attached to his ex-wife’s dad to the point where he feels like it’s his dad because he didn’t have a father figure growing up. I have expressed how I think it’s a little weird how attached he is to his exes family and that the fact that he wants to go spend weekends up there with his exes dad, when he has the kids and sometimes when he doesn’t makes me a little uncomfortable . I can completely understand bringing the kids up and spending the day with your exs dad since that is their grandparents and he was close to them but I think it’s weird that he would want to sleep there for multiple days. I have expressed that in the future I would be extremely uncomfortable if he was up there for the weekend why I’m sitting at home with my kids, and potentially our future kids while waiting for him to get back home. He says I’m in the wrong for saying I care about him but wanting to “rip” someone important out of his life. I’m not saying to cut him out completely at all I’m just asking for him to understand how weird this situation is for me and that my worries are valid. Am I really in the wrong for wanting some fair boundaries around this?