should i be concerned?

so i was talking to this guy on the phone last night. we had a summer fling and were gonna end it once college started cuz he's now four hrs away and it's his senior year but we really like each other so we r still talking and gonna visit each other frequently. (1 week to be exact :)) but anyways last night on the phone he asked me if i met anyone while i was at bars with my friends and like did i talk to another guy and have a more than small talk conversation and i said yes (even tho i didn't) and then i asked hbu and he said yes but i didn't do anything with them just talked. (i probably shouldn't believe that but that's ok) anyways after awhile we then started joking around talking about side chicks and he's like yea u always need a side!! and i was like oh yea for sure but we were joking... he likes to joke around a lot. but he's starting to tell me he loves me and wondering why i wont say it back but it's bc i cant say it back without knowing 100% if he's liking anyone else. like deep down i don't think he is in my gut bc he contacts me every night but still i would have no way of knowing and technically we r both single so he could do whatever he wants. u think its right for me not to say i love u until he's my bf? i mean i don't feel it yet but when i do.