What would you do if the father of your 4-month old child has only seen them a total of 6 hours their whole life threatens to take you to court for custody so that you dont file child support through the state because they dont want wages garnished?

Backstory- he made this threat and asked to make an agreement out of court. I called a lawyer I know to write up this agreement. He said he would split costs for it. He signed the agreement, now he is not following the agreement at all. His family has never met her and is not involved in her life at all. I am a full time working mom trying to provide for my child. He makes 5x as much I as do. He has done nothing but cause me stress and anxiety my whole maternity leave. He told me he would only sign his rights away if “my dad adopted my daughter” which was 100% weird. I am just afraid of him suing me out of spite and the court giving him minimal every other weekend and one day a week. He travels a lot and lives an hour from me. Has no car (lives in the city), no baby items, and can’t even put her diaper on correctly. I just do not want my child unsupervised with him. She had a hospital stay a few months ago, she was in respiratory distress from a virus and he didnt even come to the hospital, yet made a big deal about me asking for legal custody because he “wants a say in her medical” yet he wasnt at her birth, never came to a pre-natal appointment, and hasnt been to any of her doctors appointments. I just do not know what to do. My child does not know him at all (his choice, not mine) and when she is screaming with him he has no idea how to settle her down and it breaks my heart.