Jail time? Am I wrong?

I have primary physical custody, my sons father has visitation. Our son is 2, we have to go by our school district for holiday schedules. For thanksgiving break it was suppose to be a pick up time at the fire station at 5 p.m til the following Saturday. I made sure I was there 10 minutes before and waited for 20 mins. My sons father did not come (I live in NC, he lives in NJ). He told me that he plans to come tomorrow instead at 5 p.m however I’ve made plans early tomorrow morning to fly out to see family. In our agreement our judge was very specific and also said that “Nothing in this order shall prevent the parties from altering this schedule (including the holiday schedule) upon the mutual written consent of both parties. “Neither party shall deny the other parent their visitation at the specified exchange times of 12 p.m or 5 p.m. as stated herein If a party is going to be late to the exchange, they shall notify the other parent as soon as they reasonably know they will be late to the exchange. His attorney and him have been threatening me with jail… is this possible? They are not extremely anal with me when it comes to the agreement but want me to comply and be flexible while threatening me without any notice. Is that not a forfeit visitation? I know that it may not be a big deal to some moms but I paid a lot for a morning ticket and I’m packed along with my family so I’m a little annoyed with this.