he only says I love you when I'm leaving…

I don't remember how we started talking about it but it was a discussion about how my boyfriend hardly says I love you anymore. he says "I don't need to tell you when I'm with you if you already know that I do. if you're leaving and I know I won't see you I tell you just so you know in case something happens, but other than that I don't think I need to say it when we're just spending time together." and maybe it's because growing up in my family we didn't end a conversation or hardly ever leave a room without saying I love you to one another. also maybe, when we had sex for the first time I just felt instantly clingy and needed reassurance that he still wanted me just as much as he wanted before. he did fairly well at letting me go through this needy phase and giving me extra affection to make me feel more comfortable but now he's convinced that I need to be weeded off of it. I mean apart of me knows that we've just gotten comfortable in our relationship but still, I don't think I'm wrong for wanting to hear it every once and a while without a reason...