Guy is telling people we had sex even tho we didn’t

Okay, so I’m at my moms hometown for summer and I’m staying with my grandma, well there’s this guy and we sat outside my house and we just had a normal conversation you know, like where are you from and stuff like that well today I found out that someone is saying that me and him had sex, even tho we didn’t Im not sure if it was him because there was a couple of people there like his friends were around the corner, but it honestly just pisses me off that someone is saying that, advice?