5 year old playing with older kids

My 5 year old has asd and adhd. He isn’t allowed to play out the front unless it’s on the front garden and I’m watching. On our road there’s 3 girls - one 11 in year 6, and a set of twins around 9/10 in year 5. My son goes to school with these people and when he sees them he always gives them a hug I’m assuming to show his love/affection bless him.

The girls seem well mannered and okay. I remember one time my neighbours older kids were being mean to him and the one girl(the 11 year old) who was out at that time told them to stop as he’s only little and of course I told them too. They came to the park with us the other day and was good as gold even my son. They knocked on aswell to play on the front garden with him and his younger brother (2) for 10 minutes before they had to go in and mine to bed. Is this a problem? My mom says he shouldn’t be playing with them and that they’re too old and while I agree about the age he isn’t allowed off the garden and is always supervised by me as he is easily led and can’t sense danger(such as road crossing etc) he’s really smart for his age and says that the twins are his best friends (they’re younger sister is in his class but we hardly see her on the road) I personally think it’s sweet but I wouldn’t leave him unsupervised due to his asd/adhd and just out of fear/paranoid from my younger days I like to make sure his safe. They knocked on yesterday too and because the door was locked they gave him a sweet they bought him from the shop.

Anyway back to the question is this a problem? Should I stop him playing with them on the front garden? I know at school I can’t stop it and if it’s going to upset him I don’t really want to as long as they’re nice and safe to be around. My mom thinks they’ll try and take advantage of him in one way of another and is worried he’ll end up hurt. He’s also asked for them to play on the trampoline in the back garden too but I’m not sure how to react to that