Father's Day advice?

I planned a day for my OH for Father's Day being the first time he had his son over night and all day on his own this year. We planned to have a day out, go for dinner and give his presents. His sons mother came and collected him at 1! I think it's unfair how his son spent Father's Day with her new boyfriend and not him. Do you's think Father's Day should be spent with their father?