I’m I wrong for wanting to know where my kids are?

My ex and I have nearly 50/50 custody and the court order states we are supposed to do exchanges at each other’s residence. He was living with a family member but he got married this year and I don’t think he lives there anymore. I think he just continues to do exchanges from there but then takes the kids to his actual home. When I asked him about it, he refuses to tell me anything except that he doesn’t like that I gave his phone number and address to other people who have threatened him. I haven’t threatened him and I can’t control what other people might say to him. Don’t I have the right to know where my kids are living when they aren’t with me?

In the past I’ve had other people such as a couple of people I’ve been in a relationship with and they may pick up the kids if I can’t. It’s not my fault if they can’t get along and we should be doing what’s right for the kids and according to the courts what’s right is that we know where each other lives. So 3.5 days a week I don’t know where my kids are. Can I take him for contempt of court since he’s not following the order?

Clarification: I have his phone number but not his real address.

Clarification: the kids say they don’t live there anymore but they’re young and can’t tell me the address or anything like that.