Dad wants to see him overnight SOS

My sons dad wants to see him overnight and for the whole weekend and I don’t feel comfortable with it,especially because it may effect him feeling secure since he’s only 4 months old and he’s never even been alone with his dad for more than an hour. Also I’m with his 24/7 and only my mom has watched him for a few hours before and when I came home the next day he wouldn’t let me put him down. I want to be fair to his dad and told him he can take him for a few hours during the weekend but I want him back. He says that’s not a fair trade since I see him all the time and if we were going to do that,that I had to either pick him up or drop him off to equal out the driving which is BS to me.He says he’s going to take me to court if I don’t let him stay overnight.can he do that? And what should I do? *LO is breastfeed, also and the last time I just let him keep him the other room for the night i came in to where they were the whole room smelled of poop and LOs butt was so raw he cried when I wipped him also he’s short tempered and constantly gets upset and frustrated just for normal things like he spit up on his shirt and when he has previously came to get us he has been late or just cancelled and doesn’t help financially with him when we’re not at his house.