Coparenting, wtf am I the problem?? Am I a bad mom??

I’m honestly super pissed right now so I won’t get much into detail but I have a 4 year old son my sons father was never on child support, we basically have never taken this to court I’ve always had my son on Saturday’s, always and for that reason I’m not able to get free time and go out. He on the other hand gets shit face wasted and wakes up hungover on Sunday’s which I never take him because my child needs attention and not be watching his father sleep off his hangover. I asked him from now on to please keep our child Saturday’s because I also have stuff to do and never have free time to do it, so he starts to argue with me and says he’ll only keep him Sunday’s and that’s all I then receive a text FROM HIS MOM telling me to leave him alone because he almost knocked down his bedroom door on how he pissed he is, he still lives with his parents and claims to be the father of the year I honestly don’t know what to do up to this point I’m fed up and I feel like I’m giving him a hard time

Id also like to add that I never took this to court because of his legal status. His sister told me not to do it which is why he’s not on child support