Am I wrong

I’m married & I gave birth to our first baby 6 months ago. My mother in law and I don’t have much of a relationship because I feel as if she’s very mean and controlling & upset that I “took her son away” from her… he was 26 when we started dating….. Anywho, she has lived with her mother for more than 18 years. Me & my husband would go over to his grandmother/moms house where: his brother, sister, mom, grandmother, cousin, aunt and her 2 kids stay at. Well, his aunt has a daughter who’s 5 who’s disobedient and doesn’t listen. I’ve told him to tell them to NOT let that child around my daughter.. well one day I saw the 5 year old PICK MY DAUGHTER UP from the car seat & almost drop her. Since then, I didn’t let my daughter go over there anymore because now you’re playing with my daughter safety. His grand mother and mother was upset saying we were keeping the baby from them even after telling them our reasoning… they’re not reasonable people… There was other reasons too but I’m not going to get into that. Anywho, I have called and text messaged his other and grand mother telling her she can come over to see the baby whenever she wants. I even gave her a key to our house so that she can come as she pleases and is very much welcomed at our house. Both refuses to come.. 2 months goes by and she visited only twice.. both say that the grandchild should go to them, & that they shouldn’t come to the grand child.. mind you they live less than 5 miles away.. my mother & father on the other hand lives more than 1 hours 1/2 away and they come and see my daughter VERY often….. my mother in law also stated she doesn’t want to come over & that she wants to be able to take my baby and do as she pleases and she wants that baby at her house where again, I did not allow her to be at because they keep letting the 5 year old keep touching and try to pick up my baby even after us telling them not to.. we’ve had emergencies where I had to work and she refused to take my daughter but thank god for my mother she would come drive hours here and stay with her… my MIL’s pride made it so she didn’t see her grand daughter which honestly it’s not my problem because I tried. Anywho, fast forward his mother finally found an apartment to live at after all of these years living in that congested house with his grandmother because her and her mother kept arguing. She moved into the apartment yesterday & it is under my husbands brothers name (his mother is an immigran & can’t put it in hers) even though she had a WHOLE HUSBAND (also an immigrant) whos not his father she married 2 years ago…… anyways, she says now that she’s moved out and the 5 year old isn’t there that she will be able to have me drop off my baby there on my way to work or drop her off so me and my husband can have date night…. I was ok with the date night idea but because i feel as if my daughter is still too young, I don’t want anyone else to take care of her outside of our home. My mother has been staying with up extended period of times taking care of my daughter in the house while I returned to work 2 months ago. Anywho, todays I decided to take a drive down where she lives at because his brother made a comment about it being ghetto. Don’t get me wrong, I grew up in the “ghetto” so things like that don’t get to me. I now have a great career and worked my butt off to get where I’m at now. So I didn’t think too much of his comment until today.. I drove where she lives at and let me say she’s in the projects forsure… I saw 2 crackheads fighting over a chair at the house across the street ….. and next to it is a trap house…. NO JOKE… I also looked up the area is it’s considered the highest crime in that area….. like I’ve stated, I grew up like this and she moved to the trenches. She left a very nice surburbian neighborhood living with her mother to live HERE… I was annoying & pissed and told my husband my daughter is NOT allowed over to her house because now you’re playing with her safety. His mother was so desperate to leave her mothers house that she took whatever place she could find. Alittle back story his mother is very controlling & a narcissist. She likes doing what ever she wants and gets upset when you stand up for yourself. She also NEVER listens to reason…. I told my husband and he had the audacity to say I’m keeping our daughter from her grandmother when I in fact handed his the spare key to our house to give to his mother that I had made last week… Am I being unreasonable?? He says it seems as if I’m making excuses.