Priorities (wife and child or controlling father)

My husband and I have been together since I was 15 and he was 17 - we are now 34 and 36 (married in 2009).
His family has been a pain in our ass since day one. Bipolar mother with EXTREME moods, Very overbearing brother who has trashed our newly built (and 1st home), attacked my husband on numerous occasions and left threatening messages on our machine, and father - should actually say "man who donated sperm" who describes himself as someone who likes to play with people's minds. 
The father walked into my unlocked home last week, drunk as usual and told me off in front of my husband and  
I's 2 year old daughter. This is just one of the many occurrences that have gone on between us. 4 years ago my husband and I got into an argument that ended in me calling the police. His father called and yelled at me for "ruining his sons life" also said I should "take what I get and shut up!" He is VERY sexist and his sons can't do anything wrong. Well for example, his new wife makes his lunches, does his laundry, drives him everywhere so he can drink... He's a pig! 
Point is, when he comes to our home and yells at me, orbit he's about me to my husband... My husband doesn't say anything to him to stand up for me. I refuse to let him be around our daughter because he is a pervert and has made suggestive comments before. He called the house today and left a message for my husband reminding him to bring the "little girl" over tomorrow. My husband failed to tell me about this. My husband is now mad at me and getting very defensive  for his father. I hope this makes sense! Am I wrong? I don't feel like my husband is on my side at all! Worried for my daughter! What would you do?