Neighborhood kids

So this is new territory for me. My son is old enough now that some neighborhood kids will stop by from time to time and play for a bit. He’s very sensitive and gets upset when they decide to leave. Has anyone dealt with this kind of issue before? There is also one kid in particular who knows he gets upset and will say he’s leaving if my son isn’t playing or doing something he wants him to do. I heard him say it and told him to pick one stay or leave and stop saying it to make him cry and now I’m wondering if I over reacted.