should I stay or no?

So I've been dating this guy for four months now. He's given me gifts, we've had sex, I've met part of his family and he's coming to my birthday party and will meet my whole family at the end of this month which all sounds great except he is so hesitant to call what we have a relationship. I'm happy being with him but for some reason it's important for me to have a label on what we have... I mean we're both not seeing anyone else and we care about each other and he even told me he thinks he loves me (it was over text and I told him in person that I was pissed I didn't get to hear that in person and I felt like he took that away from me and we talked in through) so he obviously cares a great deal about me but sometimes I think he doesn't. He works a lot and has a kid which is fine but I feel like he maybe forgets about me. I really just want to be in a relationship and want more of a commitment. I just told him I'm not having sex with him unless we're in a relationship. Good or bad idea? What should I do now? Ditch his ass or wait for him to be comfortable enough to be my boyfriend eventually?!