Anyone worried their baby won’t love them ?? 😢

I’m a FTM and I have this fear that if I’m not the one holding my baby or caring for him at all times he won’t love me. It’s caused some problems in my family because I freak out when anyone holds him besides my husband. Even when I’m exhausted and our parents offer help I refuse because I’m scared my baby will think I don’t care about him. I just really want him to love and be attached to me and I’m scared if anyone else does my job he will love them over me. Do you guys feel this way?