Bad parent or good parent

I'm a mother of a 4year old amazing little girl. Her father has nothing to do with her hasn't sense day one. But that's not my issue.... The issue is that his mother and other daughter ( that he doesn't take care of) want to see my daughter. I never had any problem with it before until I found out when she was going to her grandmas her grandma would call her son (the father) and have him come over. He doesn't pay child support doesn't say happy birthday merry Christmas nothing. so therefore i dont feel as if he should be able to see my daughter. And sense grandma cant respect my wishes i dont let her or the sister see her either...Am I a bad person for not letting the grandma and sister see her ??