It's complicated

Okay so I've been seeing this guy for about 3 months now. We agreed to be exclusive and only see each other after about 3 weeks. Well through out the 3 months, he has reminded me about 4 times that he likes me but he isn't ready for a serious relationship. He dated this girl before me that freaked out on him all the time and fought every day for 2 years. But they broke up 3 months before we met. He shows signs of being effected by the toxic relationship by saying sorry often or thinking I'm upset when I'm not.

Whenever he brings up he isn't ready I tell him that's fine because I want to build up to a relationship, we are learning about each other still. And he always agrees. We go on dates, met our parents, I've met his friends recently (which originally he said would be too serious), have sex,etc. We only get to see each other once a week (twice if we r lucky) because he has a full time teaching job plus 2 coaching jobs while I'm a college student athlete.

My problem is, I have no idea if I'm wasting my time or if it will progress into a serious relationship. I'm battling with being patient and helping him be there for him support him etc. Or just leaving and telling him its either now or never for a serious relationship.

What do you think?