How do I know when it’s time to break up? Rant/advice

Let me start by saying, my boyfriend and I are madly in love with each other. We have been dating for just over 2 years and have been long distance for just over 1 of those years. We have seen each other about 5 times in our time apart and I have another 2.5 years in school. He has 3.5 more years. But the real issue is, we have different visions for our future. I want to get married and have kids. He doesn’t want to get married. He’s afraid. His parents don’t have a good relationship. But I’ve decided that I love him and don’t care as much about marriage as I do being with him. Especially at our age. It’s too soon. But he told me today he thinks we’re holding each other back. I hate that he feels that way but we don’t want to break up. Part of me thinks he’s just getting scared because we’re getting to the age where people start seriously thinking about getting engaged. I just turned 20 a few days ago and I think it’s getting to him.

He and I have been talking about having sex for a while now. We’ve had the perfect opportunity to several times but he ended up saying no. I found out today that to him, sex is basically marriage. He told me if we have sex, then that’s a promise that we will get married someday.

Anyway, long story short, we love each other. But sometimes things have to end. The thought of breaking up kills me. But I want to do what’s best for both of us. Thoughts?