Visitation schedule for 3 month old infant?

My sons father and I are no longer together due to his infidelity and emotional abuse. I had attempted to stay friends for the sake of our son and so we could develop a good co parenting relationship, but after he blew up and caused a scene at the hospital moments before I gave birth I gave up on that idea. He continued to treat me disrespectfully and pick fights every day in the hospital and every time he would come visit afterwards. So now I want nothing to do with him aside from our son. I  breastfeeding exclusively and am not comfortable with him taking our son just yet. (Not to mention he has questionable judgement and doesn't make good decisions) We had previously agreed on a 3-4 day a week visitation schedule for 2.5 hours per visit, given our son's feeding and sleeping schedule. But my sons father is complaining that this isn't enough time for him. And he complains that it's too far a drive for him and traffic is too bad. He doesn't even come most days. He is also demanding that I bring our son to his house 50 minutes away once a week or 2 Saturday's a month. I feel that's a bit much to ask as I am the one taking care of him by myself. Especially when he contributes nothing financially. So my question is what is a reasonable visitation schedule until our son is old enough and comfortable enough to go with him on his own. And am I being unreasonable by not wanting to agree to take him out to him?