Titles advice

I’ve never had a good relationship with my step father. He’s always been very rude to me and we don’t get along. Flash forward to us having our baby. He didn’t even really care to acknowledge or look at the baby when we would go over up until now. Now that our baby is 5 months and starting to get more interactive he’s wanting to hold him more and such. Now before he was born we decided (my husband and I) that he would just be know by his name and not grandpa. Because he didn’t want to be a father figure to me so we felt he shouldn’t be able to skip that and get the grandpa title. But now whenever he see him he refers to himself as grandpa. For privacy we will call him bob. So we will say “Do you want to go to bob?” When he wants to hold the baby. But he will respond with “Come to

Grandpa”. How would you fix the situation? I know some think we should just let him be grandpa, but he in no way shape or form acts like a good father or grandfather and we feel strongly about this.