When's the right time?

Okay so me and my boyfriend met last year March 2013. I had a recent break up so I didn't pay much mind but he was really into me. Until last summer, I moved to another state and he still insisted on waiting til I moved back cause he really liked me. Long story short, we made it happen and got serious last November. I took atleast 3 trips to go out to see him I'm Florida til I decided to move in with him (his IDEA) we are now living together but we've never said I love you to eachother. At first we agreed to say it at the right time and not just toss the word around...but it's been a year? When is the right time? Is he just holding back? Am I rushing?