He said I'd never break up with him.

We've been together for 6 months now. I'm 24, he's 27. He's a very attractive shorter fit guy while I'm a taller beautiful curvier girl. But to get to the point.. today in the car he said something as a joke but then said oh you'd never break up with me.. (which he definitely wasn't joking about). idk why it bothers me so much that he said that but it does. I feel like he thinks he's the most attractive man I've been with that, he could do what he wanted.. which my ex's aren't unattractive in anyway but not as good looking as him. idk.. but I instantly looked at him with a look and said " yes I would, remember I WANT you but I don't NEED you. If you would do something fucked up, I could always leave and still be okay. (I'm financially independent). I could tell he didn't know what to say so he made a joke saying "oh well then you must not love me." But for some reason hours later I still keep thinking about it.. am I being a bit crazy.?