Sons friends brother no bueno

Okay, so maybe you'll can help me with this dilemma I have, my son (3) loves my neighbors daughter (5) and they are the cutest, I love the mom too, my problem is she has a son (8) who I don't particularly like or want around my son, when he wants or doesn't want something instead of asking nicely he'll yell, push, shove, ect. even his baby sister (2) I had a tipping point, he played a joke on her where he pretended to punch her, I don't think that's anything to joke about especially if it's a boy hitting a girl or baby none the less. I've talked to the mom and she honestly tries the best she can. I talked to my son about it and he gets that you never ever joke about hurting in any way shape or form, but here's my dilemma, they won't come over to my house (we have a huge "scary" dog) I get it, I won't go to a park because I have a medically fragile daughter and parks are literally germ fields,

but that being said anyone have any ideas how can I help him have a relationship with her if I don't want my son around her son?