WIBTA for telling my son he can't eat lunch with me anymore

I teach high school geometry and my son is on his sophomore year of highschool. He has always had trouble making friends. He has anxiety and is in therapy and medicated for it. He didn't make a single friend his first year of highschool last year. He comes into my classroom during lunch because my lunch is the same as his lunch period and we eat lunch together. He did this all last year because he had no one to eat lunch with and didn't want to eat alone. I worry people may think he's a momma's boy and make it even harder to make friends. Plus he won't make friends eating lunch with his mother. I feel it will just make him seem more uncool to other highschoolers. Would it be mean to tell him he can't eat lunch with me anymore?