Should I break up with him?

Let me start out by saying that this is a very long story but I would be very grateful if you would read it and give me some advice. My boyfriend and I have been dating for about six months. We're both twenty years old and met at school. We started dating in the summer when we were both free of any commitments, which made it easy to be together all the time. After dating for a few months, he came on my family vacation with me, my parents, and my brother. I went to his house out of state and stayed with his family for a few days. Our parents had dinner when his parents came down to drop his younger sister off at college. I thought things were moving fast, but I was happy about it. Once classes started up again, we knew it would be harder to spend time together but agreed we would work on it. Classes had only been going on for a month when he told me he thought we should just be friends because "I never wanted to do anything or go to school events with him." I was completely shocked and caught off guard. I tried to explain to him that I was still adjusting to school but still wanted the same things as him. He insisted we break up. I was absolutely crushed. I knew things hadn't been perfect lately, but I thought we were still at a good place and thought I might be falling in love with him. We stayed away from each other for one week. I tried to convince myself that it was for the best but it was very hard. One week later, I decided to drop his stuff off and talk to him because we had been planning to go on a trip across the country together with our friend and her boyfriend for fall break, and I was really concerned about what was going to happen with that. We were going to leave almost exactly one month after he dumped me, which didn't give us a lot of time to become just friends, which is what he said he wanted. So even though it had only been a week, I met him and talked to him. When I saw him after the breakup, he acted like he made a mistake, but I thought I was just imagining it because it was what I wanted. He ended up telling me how sorry he was, how much he missed me, and how much he wished he could call me to tell me about his day. I was so happy that he was giving me another chance and that we could go on our trip the way we were meant to go. We talked about our issues and why we broke up and how we could be better partners for each other. This was one month ago. The last month has been difficult, but it's also been fun and nice to be with him again. Sometimes it feels like we really get each other, and other times we sit in silence with no spark. It's one month later and we got back from or trip yesterday. It was fun but I think we both feel like the spark isn't there sometimes and it might be time to go our seperate ways. It makes me so sad to think that I might not have him in my life anymore. We are either very happy or very confused about what we want lately. I still think there could be a chance that we could work things out but part of me feels like it's time to let my first almost-love go and hope I will one day find the love I've  been waiting for with someone else. I don't know what to do and it breaks my heart to think about letting him go.