Can she legally stop us taking his son?

My partners ex wife is being an absolute c*nt lately and has stopped us from seeing his son. When they split up, my partner agreed to his ex taking his son with her back to her home country as he was only 2yo, and a list of conditions were informally agreed upon. One of these was that he had regular contact (Skype calls everyday) and another was that their son would be able to come and visit his dads side of the family for at least a week, and at least every six weeks. It was later agreed that school holidays would be divided equally (so we have him for Christmas one year, he stays with her the next year, 3 weeks each during the summer 6 weeks off, then take it in turns with every other holiday) and this has worked out well for the past 2 years.

We agreed months ago that we would have him over with us for the first two weeks, plus the last one week, of the summer school holidays. She recently asked for money to buy him new shoes, and we refused as we had bought him a pair plus a load of other clothes the week before while he was visiting us, but she threw them all away when he got back to her because they “looked cheap”. After a load of abuse about how neglectful we are blah blah blah we get told that he can’t come over to visit during the last week of the summer holidays because her husband has been given that week off work, and they’ve booked to go on holiday. We’ve also booked time off work and have made plans with our family to go on a camping trip during that week. I tried to rearrange everything but work can’t be that flexible and some of the family would also miss the camping trip.

Question is, can she legally stop us from taking their son and bringing him back to stay with us for a week? There is no custody order in place (it was totally left out of their divorce), they have joint parental responsibility, and there is also no residency order in place.

Any advice appreciated