Sick and tired

So I’m currently going through some childish bs with my boyfriend. He’s trying to hold me accountable for other men who are trying to get my attention. No I haven’t been entertaining anyone because I am head over heals in love with my boyfriend. There have been men that have tried to split us up by telling him that we’ve messed around which is not true. The only thing I was guilty for was speaking to these men on a friendly level when they tried to say something out of the way I stopped them in their track and let them know that I’m spoken for and I would stop communications with them. Even though I’ve let them know that I have someone they still continue to try and once they see that he’s writing them to let them know to leave me alone that’s when they’ll try to say we had sexual encounters and he believes them. Now mind you females that want him have also said the same about him. They would literally hunt me down on social media and write me but I never once doubted his word. I’m spoken to him about how the relationship shouldn’t be one sided and that since I’ve never doubted his word he should be able to do the same with me. I’ve said it time and time again that others will always will what they can’t have and will try everything to get it but he doesn’t seem to see it. Now I’m in thinking that maybe what the females are saying is true because why else would it be soo easy for him to belong a stranger. I’m really trying to fix this but I don’t think it’s fixable. Anyone has been through this before? If so do you have any advice for me?