Am I crazy? What should I do?

To give a little background: I've been dating a married man who claims he is polyamorous for 3 months. I've met his wife and kids as well. At first, I viewed the relationship as something very casual. Sex is amazing and we both got a long very well. The more we talked, the more we started falling for each other. He is one of the sweetest and most genuine guys that I have dated... But the clincher is knowing it can't be anything more than just casual. He's told me that if it was possible, he would make the relationship into something more. I'm at a point where I want to tell him I can't continue dating him since I know there is no long term future... But then a piece of me wants to ride it out a little longer since I love and care for him