How do I move forward.

Me and my boyfriend have a 3 year old daughter dated for almost a year split for 2 and now have been back together a year. We have never said I love you (Well I did once when I was pregnant he didn't say it back so I never said it again) I want to be with him but I just feel like thing will never go more than what we are. I've talked to him about this and how I feel but he just said he is moving slow with it because he's always got hurt in the past when he did love someone. But you think after being in eachother lives for 4 years love would be there right? Like how long does it take to love someone. I won't let myself love him like I want to because I know he doesn't love me. What the hell do I do. Like I'm happy but now how I want to be plus our sex life has been pretty much nothing. We've had sex once in the last month. I just don't know what to do anymore.