Am I overthinking?

So I’ve recently connected with an old friend and we are trying to pursue a relationship. We talk everyday and we lived in 2 different cities in the beginning but for the past month we now have moved back to the same city. He constantly wants to make plans so I clear my day to see him but either something comes up or he just doesn’t message me when it’s the time he said we’d see each other. It’s happened about 4-5 times now. He’ll apologize for it or say he’ll make it up by rescheduling but same thing always happens. We’ve had sex before and it felt passionate and he’s always talking about that he likes me and things like that. I like him but don’t know if I’m wasting my time or just give it time.