Overnight Stays with Father

Hello all, I am a FTM & my LO is 6 weeks. My child's father & I are not together because he is immature & disrespectful when he does not get his way. Anywho, LO's father has been begging me to keep him for a weekend, but I simply cannot imagine him staying away from me overnight. I miss him so much just when I run errands and leave him with my mother for a few hours. The fact that my ex is also pretty irresponsible does not help. He also has never kept a baby before so when he does come over he has a hard time calming LO down and he blames it on my breast milk. I have to remind him to change him & that he may be hungry ... He also did not sign my son's BC, but was there for the birth. I initially gave my son his last name, but after he said he would like a DNA test before signing his BC before I even left the hospital. I had his name changed & him removed from LO's BC immediately. I told him once we did the DNA test then he could be added back since he was unsure. Fast forward to now, he also has a very bad temper & thinks he is always right. So at this point he has been forbidden to come to my house due to his last visit when he got angry at me & began yelling and cursing at me with my son in his arms scaring him. He got angry because he said that my breast milk was giving him gas & that's why he was crying. To which I responded he was crying because he was hungry & you would not give him to me to feed because for some reason unknown to me you have a problem giving him to me when you are here. Now, that I've given a little background, my question is .. At what age should I allow my son to stay with him overnight ? Or what age did you let your LO stay with his father overnight ? I honestly don't want him to stay with him overnight until he can talk, but I also don't want to be unfair ... Even though he is disrespectful to me. SORRY SO LONG.