How to break up with someone you aren’t even dating

Ok so, I went out with a guy who was not my usual type. We hit it off on the first date and hung out a few more times. I am physically attracted to him but he does small things that are just a turn off to me, and he’s an aggressive person. He’s really attached to me, and called me his gf and we aren’t even past the third date. How do I end this without really hurting him?