Am i wrong if i am bothered by it??

Guy im dating ive known for 22 yrs. We always have feelings toward each other.and recently decided to act on those feelings.He says he can see marriage , kids the whole 9 yards for us . talks about how much he cares for me. But arent officially a couple. He wants to be 100%that we're compatible . on our first date , we were talking . kinda catching up . i asked him if about any recent relationships . he told me about a female that was the most recent besides me. He admitted she was married and he knew she wasnt someone he was gonna stay with. But confessed thhat he was in fact infatuated with her. At the time, didnt bother me . but its been a couple months and i just found out they work together. He says they havent been together since we started dating . but that he was still a lil infatuated with her. He then reassured me he didnt want to be with her. Now i dont know how i feel about that? Should i be upset, or indifferent ? Do i say something . now i feel as if he is trying to convince himself to stay with me . almost like he doesnt want to hurt my feelings so he stays with me. All the while lusting after her. Im not a needy broad , i told him to always be honest even if he thinks it will hurt me. I dont want someone that doesnt want me. Cuz i find out later that he never wanted to stay itll be so much worse. I dont know what i should do? I feel like giving him space. But every time i do , he calls me asking why i havent been calling him. Im getting mixed signals help?