Advice, anyone :(

Okay so me and my SO have been dating for about 5 months now. I obviously think he is attracted to me but lately he doesn't want to have sex or do anything of that nature. He said he's too stressed out from work, but what guy doesn't want to have sex? This has happened before and I talked to him about it and he said he doesn't want to base our relationship off of sex. I hate to make this about me but it really makes me feel bad about myself because in the back of my brain I just don't think I'm good enough for him or something and/or he's getting bored of me. What do you think? What should I do?