Not sure what to do

I have a 10 year old previous relationship and a 4 year old my ex for my 4 year old has been in and out of his life we never live together we were friends and end up dating he has other kids I just found out 5 other kids were only in our early 30s he never spent a dime on my son every time he comes over it to hang out for a bit or he wants to sleep over he stays with a friend and does not have a stable job I got sick of it and told him not to come back around my kids he disappear for 6 months and came knocking at my door at 11pm cause he had no were to go I did not answer him so he left he moved back in with the friend and wants to see my son I agreed but now I don’t feel like letting him see my son I don’t want my son to see him and then he disappears again he also wants to see him at my house cause he not bring my child to his friends plus I live really far but I don’t even want him in my house I feel like he would get comfy and don’t wanna leave i really dislike how this guy does nothing for my kid but I don’t want my son to hate me cause I took him away from his dad but