Toddler overnight stays with dad

My son is 17 months old and he only had a long period of contact with his father twice. The father wants to do overnight several days a week with him but my only concern is that my son does not have a bond with his father, he barely knows him, I told him that for our son is better to do one overnight a week and the rest of the day short visits so that they can build a relationship/secure attachment before having the toddler overnights. He’s not accepting this and I wanted to know if Im being unfair by doing this?