"If you want"

So I'll ask my bf if he wants to call or hang out and he keeps saying if you want. I know hes not meaning it this way but I dont like the phrase if you want because it makes me feel like he doesnt care if we hang out or call. Idk just the phrase If you want makes me feel like I'm not being wanted there because he never says I want to hang out with you or I want to call you and it just upset me. What do I do?