Am I in the wrong?

I’ve taken care of my niece since birth. Her mother has had a drug problem for years. My brother has stayed with her until three years ago. My niece has lived with me for those three and my brother for 1 and 1/2. He works full time so he left my niece here with me and goes to the school down the street from me. I do everything for her , washing , helping with homework, meals, when she’s sick it’s me. And my husband drives her to school. She sees my brother after school for a few hours. Well now we have been quarantine. I said if he showers and changes after work he can come over to visit. But I said if you take her she can only stay at your apartment. He disagrees with me and says that his daughter needs to go out and about. I’m like no there’s a virus. He says I’m being mean. He is also upset I won’t let her cousins come over. I have a low immune system because of an underlying condition he knows this. I say if you want to keep her at your house that’s fine. But I won’t be able to watch her anymore. He is very angry and said I’m taking his parental rights away. I say how? you can do what ever you want. But I have rules in my house. My husband agrees. He has stated the same thing. My brother says that there is nothing wrong with taking her to the stores or to his girlfriends house where there 6 other people there. The girlfriends mother works in a health clinic. And now the girlfriend was sick and just got tested for corona. But he feels it’s ok to take her there. He says he’s the parent and he can do whatever he wants. I say yes you can but unfortunately this is serious and I would like for u not to take her but I can’t stop you but you will have have somebody else watch her. Am I wrong or am I taking his rights away? I’m really not understanding him.