Rights pulled? Deadbeat father.

So, my ex left me when I was 12 weeks pregnant for my son.. he didnt come meet him until my son was 6 months old for 3 days (I live in NY and he lives in TX) he messages me "how is he" every 3 months- barely pays the 20$ for child support once a month. (Hes late 5 months now)

I took him to court last year for sole custody and it was granted to me. He has 2 visitations a year (his request)..

He hasnt seen my son since he was 6 months and he is almost 3 now. Im married now to an amazing man that has been a father to my son for a year and a half now..

Could I take him for his rights if my DH wants-to adopt and take over parental rights?