Custody/adoption advice

It's long but I'm going to try and summarize 9 years, so bear with me and my typos please.

My youngest daughters father and I dated for about a year when she was conceived and at the time we found out, let's just say the relationship was holding on by a string. We tried to make it work but it wasn't happening finally called it quits right before she was born. Well she's 5 weeks and he takes off to a city 6 hours away. I'm like seriously. Whatever. I move on being a mom, I mean what else do you want me to do. At 3 months she was hospitalized and ended up having two surgeries one at 4 months one at 5 months. And he is no where to be seen, none of his family even showed up and they lived in town. Yeah I'm upset but again I let it go. I have a sick child to take care of.

He doesn't see her again until she 1. And somehow it's my fault because I refuse to drive 6 hours away with my sick/recovering daughter so you can see her. Um not my problem. On top of this he is living with his alcoholic cousin and his wife plus 3 other couples in a 4 bedroom house. No job, selling and smoking weed plus who knows what else. All of this is what he tells me not first hand knowledge. But seriously what kind of parent would I be to look the other way just cause your her father and put my child in a drug house as he called it. Hell no, sorry not sorry and could care less. So he's to broke to come see her and doesnt see her until she's 1. At her 1st birthday party it comes out he expected me to eventually get back with him and move to be with him. His whole plan was to move first and have me follow after he got settled. Excuse me, no. You had a fantasy in your head that didn't play out in reality and now I'm the bad person. GTFO. Him and his family at her birthday are calling me ungrateful and a multitude of other things. Of course my family at this point is like leave, and makes them leave.

Long story short for 4 more years he pays no children support maybe sees her 2 times a year and it always my fault because I'm not wasting my money to drive down there. Did I mention his own family only sees he once a year and they literally live 5 minutes away. They'll drive past my house to go to weed man but driving to see her is too far and why can't I meet them halfway. By the time she is 5 I have an opportunity to move out of state and do better for us. I take it. All of a sudden he moves back and takes me to court because I'm moving. I tried really he had her every other weekend and everyday after school when he moved back. So we go to court, he fails drug test and gets put on child support and drug says I can move and do what I need to do because he basically abandoned her welfare to be completely in my career for 4 years and has no grounds to question them now.

So I move start my new job, bought my house and my kids are thriving and happy. No court order but I send her to him the first summer I'm here and I had to call my father to pick her up because he calls me begging for money because he can't buy groceries and his lights are getting cut-off. I tell my dad pick up my daughter, he does and then he begs my dad for money. My daughter came home giving me a tutorial on how to roll a blunt. Oh did I mention I had to go court against the state because after she was home he went and filled for welfare using her and my oldest daughter and the state tried to come after me for 10,000 they paid him. I won of course because my oldest isn't his and they were both living with me. Like WTF! I put my foot down. No more I'm done. I met my husband first year I moved and we're now married with a baby on the way. In the 3.5 years we've been together the only time this man called was Xmas, her birthday, when I got married (literally called calling me a cheating B etc etc. I'm wrong blah blah, again haven't been together in 6 years at this point and he has a live in girlfriend, but I digress). Never sends birthday or Xmas gifts, we don't expect them but he tells her he's sending it. He only pays 133 a month. After my marriage took or tried to take me to court to get him off child support.

Of course judge said no. Somehow that's my fault. Well since I've been married ( more like since a year after moving) he's disappeared. No calls, he's 6 months behind in support, etc. So I'm done I'm not a bad at least I don't think I am but I'm just not a nag either you want to be in her life or you don't. It's not my responsibility to make you parent. I don't feel I've ever kept him away from seeing her, but I really don't feel it's my job to spend money and take time off my job and life for him to see her. It either he expects me to pay for everything so he can see her and refuses to even meet me halfway.

Well now my daughter wants my husband to adopt her like he is my oldest daughter. And when I tell her she has a dad so he cant legally adopt her. She gets upset and says but he's really not my dad.(the ex) I didn't know him until I was 4 I'm about to be 9 and where is he. This is my dad and I want my dad's last name. Between her and her older sister (12) it's a battle. Now neither wants to be adopted unless they are both being adopted. So hear is my dilemma. He won't sign over his rights because he thinks he's the AL Pacino of fathers, and he hates that I dare let this man(my husband) into his bed and his daughters life. Again his words. So I really don't know how to proceed. It's been 2 years and the girls aren't backing down but legally I can't make him give up his rights even though he hasn't been involved sine she was 5.

What should I do? Go to court? File abandonment? I'm just stuck. Really he wants nothing to do with her unless you cater to him, throws a fit when he doesn't get his way, only time he calls is to try and get me to leave my husband for him or to make his oh to breakable promises to her. I'm just done.

Mini update: ...(just found out he has 2 other kids(younger) that he has not paid support on. Talked to one of the mothers and she's going through a similar situation as of did earlier on with him. Just sad a frustrating