Is the feeling real or just a feeling

To make a long story short. My boyfriend of 4 years broke up with me about a year and a half ago. He and I had committed to waiting until we were married to have sex. Found out he had been with multiple girls at school. Forgave him. Then he broke up with me. Feeling unwanted I then lost my virginity to some random. And have been with 17 people since. I had hit a low point then met someone who I thought I loved and who I thought loved me. He then broke my heart as well. Since we broke up i have been keeping to my self which has been about 6 months. I started talking to this guy New Year's <a href="https://play.google.com/store/apps/details?id=com.glow.android.eve">Eve</a> and we spent every day together for the next week. I took him back to school. We talk on the phone every day, face time, text, ect. We have spent the past weekends together since he goes to school a couple hours away. This past Sunday when we were napping he mumbled I love you but when I said what he didn't respond. I didn't move for a few minutes. Then I woke him up and told him I needed to go home. He was clearly asleep when he said that. I guess I seemed upset because he kept asking what was wrong. He thought I was mad. I wasn't I was just scared because I almost said it back with out thinking. I am going to see him tomorrow after a meeting for work I have close to his school. And I'm going with his family to his wrestling tournament on Saturday. I've been considering not going because I feel like it's all happening so fast and I just don't want to get hurt again. I guess I am just curious for some stranger advice because 
My family- all still holding expectations that live up to my ex from 4 years.
My friends- just don't want me to get hurt so they don't think any guy deserves a chance
Opinions and comments are greatly appreciated.