Am I wrong for this? Just want to see ...

My daughters grandpa (her dads dad) said to his son, that I used him to have a black baby... He also said that I an lazy and lay up (my baby is 7 weeks old) ... Because of this I dont want him around my daughter.. Quite frankly I think he can kiss my ass. Am i wrong for wanting my daughter to not see him?