Court order

So there is a court order that has been in place for approximately 4 years for my ex to have contact with my daughter.

We had gone a little bit out of the court order (he never lost time) but it was to fit around his work, but he demanded the agreement got completely changed, which I did offer the same amount of time he currently has with an extra night but so it would fit around his work.

He refused it and said he would rather stick to the court order.

Not a problem.

There was a bit of drama when him & his girlfriend broke up, but that is irrelevant as it’s a he said she said thing. The court order that is currently in place means I do not have to have contact with my ex which due to recent drama suits me perfectly as I am fed up of lies and shit stirring.

He has asked to see my daughter on a day which is not his, and I said no, (I had plans) he has kicked off saying he is going to take me to court over it.

Can I actually get in trouble for sticking to a court order just because my ex is disagreeing with me for sticking to the court order?