What do i do?!!? Advice plz

I've been seeing this older guy and from the beginning we made it clear that it was going to be casual because I had just gotten out of a serious relationship. Fast forward two months and he seems be getting pretty attached. Granted we do hang out a lot since he's smart and sweet when he wants to be but other times he's just a dick and I can't exactly picture him or this relationship as my end goal. I mean I can but I'm not sure that's what I want and that I'd be happy. I know he says there's no pressure and that is casual but then he points it how it's like a relationship and I see how it is but that's not what I want. He "jokes" around about meeting my mom and our future kids and us getting married since while it's not a terrible thought I'm just not to set on this idea of a relationship with him.

I don't know what to do. I really enjoy his company again when he isn't being a huge dick, and he's considerate most of the time and he's smart he has his shit together he has goals and ladies the sex is amazing. I went from rarely being able to come in the past to now coming two sometimes four five times in a night. He's so good at the sex like I'm addicted, but I feel like he's getting super attached and while I am looking for "the one" I'm not sure he's the one.

I guess my point is I'm not ready to stop this thing we're doing but I recognize he's becoming attached and I would want to have some sort of relationship with him afterwards but I don't know how to break up with people especially with older and conflict prone/confrontational/aggressive men.