Dogs And Neighbors Who Complain

So let me give you all a little background. I have three dogs, a Siberian Husky and two German Shepard's. One is a puppy. When my neighbors moved in four years ago, they were aware of that. They have two little girls who are afraid of our dogs, even though our dogs don't bother them. We have a gate separating our backyards. They come outside and run inside screaming when they see our dogs or ask us to bring them in. If they're out for long, it's not an issue but if I just put them out, hell no we can't. My dogs doesnt bark or bother a soul. Am I wrong for thinking that they should just shut up about my dogs?