am i in the wrong???

my son's father has been in and out of his life since he was born... before he was even a month old, he spent 2 consecutive weeks of his life not seeing him bc he was mad at me... then just recently he didnt see him for four months. he just came back in his life in the middle of october, and has been seeing him every saturday when he gets off.. sunday, im going an hour and a half out of town, so i asked my dad to keep my son on sunday and sunday only.. his father just called asking to keep him saturday night overnight and not let my dad keep him.. he got mad and hung up in my face because i told him no.. for one, i already told my dad that he could keep him and two, he hasnt even been with him long enough for me to let him stay overnight yet... am i wrong for telling him no? my son is 9 months old, btw