help... 3 month old question

Am I wrong for not wanting my baby's father to take my son from me for a day? He lives TWO hours away from me, where I wouldn't be able to get to him quickly if something were to happen and also hasn't been around him enough for me to really feel comfortable or trust him having him without me. My baby is only three months and I feel him being that far from me for that long will upset him. Just need outside opinions.