5 month old

My sons baby daddy rarely has his son, he’s always out drinking. Or when he does have him he sends him to his grandpas for the day so he could sleep. I’m scared that when he starts taking his son more often like he says he will, my son won’t feel that much of a connection or whatever and freak out when it’s bedtime. I can’t go over there at night to help him either, it’s not that I don’t want to it’s cuz I can’t. My son already freaked out on my mother 2 days ago at bedtime, and he sees her everyday. Just scared he’ll be crying for hours with his daddy. If I send a shirt of mine over with him & he puts it over him do you think he’ll smell my scent & calm down? Or is that a newborn thing lol.