Should we just stop ?

So I’m 21 and I’ve recently been talking to this guy that’s 20, we met in high school. I’ve seen him about 2-3 times, late at night. He drives his moms car. Just now we were talking about having sex.. but he doesn’t have his moms car to pick me up. I don’t have a car and I’m not taking that 3 hour commute on public transportation to him.. and even if I did do that, his step mother is home and we’d have to hide the fact that we’re having sex. At the end of summer, he goes back to college in the next state over, but he swears he’s always in this state visiting, it’s only about a two hour drive and he says that’s nothing. It just seems like we’re at opposites right now. I’m thinking it’s best to just leave things alone. Idk. I do have a tendency of doubting my relationships very early on at the slightest inconvenience because I’m scared of wasting my time.

-this isn’t meant to be a “summer fling”.