What would you do

If your child's father was in and out for the 2 years of your sons life then finally quit coming around for months would you just not let him see the child anymore? He has really never helped financially either.