What would happen

My husband and I are divorcing. He knows this is a possibility but he says I’m all talk and won’t do it. Well guess again buddy. Before I got pregnant, before I even met him, I slept with his brother. I was not aware this Was his brother, drunken night with some friends and it just happened. 2 years later I met my husband And I learned he was my brothers husband. I didn’t tell him because we had agreed not to tell each other things about previous sexual members, we agreed anything before us doesn’t matter. His brother told his aunt after my husband and I got pregnant which resulted in his family disowning my baby as well as my husband. I paid over $1k for a court certified dna test to prove it was my husbands. My daughter is 4 and autistic now, her dad was only in her life for 3 months, when she turned 3 months he left. He’s never cared for her physically, financially (no child support), etc. he’s never came to drs appts. I told him that she had autism and how the dr would like for him to attend her therapy and her other appts so we can rebuild a relationship between them in a healthy way. He refused and so did his family. I filed papers after we separated and they couldn’t find him. So nothing happened. Now we know where he is because he filed a case on someone else. When I asked him to sign divorce papers and custody papers he refused saying he wanted to get back together and he would only agree to full custody or joint. I told him that my attorney and I have discussed it and it won’t happen because my daughter doesn’t know him and because he has never been involved and refuses to. He says he’ll take joint or full custody and accuses me of jumping from man to man (which is lies, just because I’ve had sexual relations with men doesn’t mean anything, those men never met my child or came to my home), and he says I move too much, which is partly true but that’s because I’m a single mom and if a landlord sells the house or my job moves me or something, I can’t help that. But I’ve been at my current residence for a year now and my attorney said It’s baseless because he’s 30 and lives with his grandmother and uncle in a 2 bedroom house, there’s no room for my child there and he refuses to get his own house. I can’t just file for divorce because my state requires custody of all children to be established with divorce and I need custody established because it’s affecting her SSI and school is wanting it. In my defense about the brother situation, my husband has slept with plenty of his brothers girlfriends and his brother has slept with my husbands ex girlfriends. They both have a problem with this I learned. And when my husband learned about it he wasn’t mad about it because he knew I loved him and knew it was years before us, his family though took his harder and they were mad at me saying that he had no idea I would ever meet his brother, but that makes no sense because I had no idea I’d ever meet my husband. And the reason my husband disowned my baby is because his family (we are Hispanic) they said it would cause problems and look bad at the church at they go to. They told my husband that it was his brothers baby and even made claims that She was another mans baby in town. Which upset my family because I’m not a bad girl and I’ve never done anything like that nor would I. I am very faithful in my marriages. And when I proved it was his baby and I was faithful they said it was too late to let her be apart of the family because it’s too confusing.