Sick Brother

I have a 27 yr old brother who is mentally ill. He was dropped on his head when he was 6 months old and hasn't been the same. He gets seizures and has the mental capacity of a child. Sometimes he is very out of control. He was threatened to murder my dad(he raised him alone) he has hit my sister, punched her , kicked her, kicked her husband down there, cussed us out, has hit almost all of us siblings, he has episodes where he will call the cops and say we are treating him horrible. This is only a part of all the things he does. He does take medication for his health problems but he's just getting very bad. My dad does not want the state to put him away in a hospital, because he just doesn't think like us. Well the thing is my sister who has been taking care of him for a couple of months now is getting married. She wanted a part time job, to help out with her wedding planning so she found one close to my house. She was going to drop of my brother and pick him up everyday. I agreed because he would be asleep all day from 5 am to 2 pm until she was ready to pick him up. Now my husband started saying that he doesn't trust him around my kids, that I'm making a mistake because I'm not going to be able to sleep in because he will try and get out of the house. That how can I agree to watch him when he is a grown man who still acts like a kid and all this other stuff. I told my sister I couldn't watch him anymore and she picked him up right now and left. I feel so horrible because he's also my brother and I wanted ti help out my sister in something. I don't know what to do anymore. I don't want to be arguing with my husband because he's partially right but now I feel bad on not helping out my sister with him.