Boyfriend wants to come with me on visitations with my ex

My ex has supervised visitation with our son at the state prison. He was already on supervised visitation because when our son was 8 he left him home alone to go get high and our son got hurt. So he's been on supervised visits. He went to prison for drug possession. I don't really want to hear peoples opinions about whether or not a child should be visiting a prison. It's in the custody agreement so I have to abide by it. Our son is 12 now and every other Saturday he has visitation with his dad. Right now they've been working on building this Lego Harry Potter set together. They keep it in the visitation room and nobody messes up with. My boyfriend wants to come with because he doesn't want me being around my ex without him because he finds it disrespectful but my ex doesn't want him there because it's weird. Advice(