I asked for no visitors

I don't want visitors for at least 2 or more hours.. Seems reasonable, right? Well my brother who is an alcoholic... Was drunk BIG TIME the MORNING of the 4th of July and I was telling him I don't want anyone there and he said "if I want to see my little niece/nephew I will be coming in that room no one will stop me, I used to work there they won't care!" That pissed me off so so sooo much!! Well later that day my hubby was telling him he needs to respect our wishes and not be so rude about it bc it made me so upset. My brother tried to fight him!!! Is it that bad for me not to want anyone there?? For him to sit there and be so drunk all day everyday I don't even want him holding my baby!