What do I do? (I know it's a long read but I could really use some advice)

I met him when I was five, had the biggest crush on him growing up. Then he moved away and i didnt see or talk to him for eight years. We're in our mid 20s now. A mutual friend of ours recently got married and we re connected at the wedding. Sparks flew right away it seemed almost too good to be true. It's been a few months now and since then we've been sleeping with each other, had breakfast with both his parents, I've known his mom as long as him shes always loved me. I've met all of his friends and we all get along great. We run errands together, laugh, talk and learn about one another constantly. We call each other baby, babe, love, hun.. We've never said we were dating but we act like it and anytime we go to his friends they say we are, neither of us correct it. Last night after fireworks we went back to his house. We were sitting on the couch with Netflix on and his head on my lap. I called him my man in one of our conversations and he went on saying he likes what we have. That he knows he calls me his lady and I call him my man but he doesn't want a relationship. He told me "hes not the one.. that hes a robot." He has OCD and he likes things done a particular way but I know hes not just some emotionless person. He has told me that it's hard for him to let people in, to accept help because he has always taken care of himself. He shows affection in his own way, not much of a cuddler or hand holder he wasnt used to being shown affection, it was hard for him when we started talking but he shows it in his own way and I've grown to love it because it's part of what makes him HIM. He's sweet and respectful and caring and he gets the cutest look on his face when he's excited it just warms me up. The last few days I've been looking at him while doing little things and just thinking "wow I think I'm really starting to love this guy" ..I didn't tell him.. Now I feel kind of crushed. Like in his mind he has set an expiration date on our time together. I'm not sure what to do now, he brought me home this morning before work and I can't stop thinking about it. Hes told me quite a few times over the last few months that if I was to walk away, he wouldn't be mad. That he knows hes hard to be around and he's gotten used to being too much for people. Could he possibly just be pushing away because he doesn't think I'm going to stick around? Am I not enough for him, am I too much? How do I process this.