Trying for full custody? Maybe...

My ex and I have a 4 year old son together. We broke up over a year ago cause he was abusive and cheated. He did good for a while but now he is never with my son when he is supposed to me. He always drops him off with grandma (his mother who is our babysitter) to go visit his new gf who is still married. This has been going on a while and my son is starting to notice dad is never around. I’m trying to get proof of this but it’s been difficult cause he hides it so well. Would it even be worth trying for full custody? Right now we just split up our time 50/50 without the courts involved but now that my son asks why dad is never around I feel I need to do more. I’m not looking for his money, but would rather my son be with me than grandma if he isn’t gonna be around...