"Not Girlfriend Material."

Today I was talking to this guy I'm kinda close to about my latest break-up and why we broke up. So, we broke up mostly because of summer but we got into an argument first about how he accused me of cheating and doesn't trust me. So, this guy says, "Yeah, when I met you I thought 'she's cute and pretty but, not girlfriend material'." He said this because apparently I 'flirt with anything that walks.' In the past couple years I've been told this kind of phrase many times because I have a kind personality that is 'flirty' but, I don't mean to flirt in any way I just am being myself. What should I do?