frustrated with my boyfriend

So I only see my boyfriend of two years on weekends. Last week I was able to see him Thursday-Friday. We only had sex once on Thursday, and that was it. I confronted him about this & he told me he's been thinking about school & college football, and hasn't even masturbayed. He told me that even if he does maturbate, that doesn't change anything with us having sex, and that I'll always he more important than porn. He also told me that I shouldn't worry because we're preparing for the future, as when we're living together we won't be doing it every day. I completely agree with all this. It's just that when I'm away for the week & he masturbates the day before I come over, and we don't do it the whole weekend, I do get frustrated bc I think I'm being replaced. I know I'm not...but how do I get reassured about this??