My mother did this, she withheld us for months at a time because she didn't like my father's girlfriends, she was still bitter over their messy divorce and my dad's infidelity. She withheld us when my dad couldn't make his child support payments. My dad couldn't ever take her to court over it either for the same reason he had issues with child support, he didn't have a great paying job and ended up with a lot of bills from the divorce so he was barely afloat and had to move in with his mom. She would threaten to send him to jail for missing payments if he took her to court for withholding us from him. She filled our heads with lies about how he treated her during their marriage, she made us believe he was neglectful and didn't care about us which is why he never showed up (when in reality she wouldn't let him show up), she used as pawns to get back at him. I missed out on years with my father and that was really painful for me, I was always by his side as a young child before they divorced. It's still painful since I don't have a relationship with him like I would have if she didn't do that. She thought that if she made us hate him or think that he hates us that it would make us closer to her when in reality it drove us away and we don't speak to her anymore. 

So I just wanted to give you guys my experience and say that no matter what your deal is, don't do this to your kid. You have no say over what goes on in the other parent's house or who is around unless something bad or a sex offender is around, so don't be petty over it. Move on and do the best you can to raise your child and don't drive away a good father that cares for his children. Some children aren't lucky enough to have a father that wants to see them so a good one that is trying should never be taken for granted.