Unfair?

My 10 year old son and his 2 friends that are brothers can be wild sometimes, they’re boys! But sometimes they step out of line so I tell them if they can’t hangout nicely then they can’t hangout so I tell them that’s enough for the day and we will see them tomorrow. The boy’s mom isn’t ever home as she works so if her kids are the ones causing trouble I send their mom a text letting her now what happened. She gives them a talk on the phone and that’s that. I give my son a talk after pulling him inside but the boy’s mother never tells them to stop hanging out for the day if something happens. It’s not just little things, for example my son started an argument with them and he got heated so I told him to apologize and go inside. However the other kids are upset now and asked why should they be punished if my son was the one who did wrong. My son is their only friend in the neighborhood that they have to play with and by telling him he can’t play isn’t fair to them. I didn’t think of it that way but I feel like if my son can’t be respectful he needs to take a step back. Am I being unfair?