guy trouble

I've been feeling down on a situation I need some advice on. Basically I met this guy on July 4th at a barbecue my friend invited me to. Ever since then me and him have been talking strong. I really like him a lot we even said I love you to each other. I want to be in a relationship with him but he tells me he's not ready. My issue is okay I understand where he's coming from he doesn't want to rush into anything just yet. However, he's saying he loves me we spend a lot of time together basically on the phone half the day. We never use protection during sex and sometimes when he gets upset with me he says well I guess I'm single now I'm just like wtf. You want or you don't; plus he told me he still have feelings for his ex girl friend supposedly that was his first love. I was upset when he told  this and it really hurt my feelings because I figure if you still are in love with someone else why pursue me. To make matters worse she's apart of his circle so when he's hanging out with his friends she's there and that's makes me uncomfortable. I'm not a insecure person but I just don't want to be looking stupid. He tells me he doesn't want to get back with her but for some reason I don't believe him. My question is do you guys think I should try to make it work or let the situation go. I'm so confused because he genuinely seems like he cares about me.