Should i be worried?

Me and my boyfriend haven’t been having sex. In the past we used to fight a lot and he always used to say that was the cause of why we didn’t because in his eyes if I’m acting ugly it makes me look ugly (if that makes sense) but here lately I’ve been working on it because I do let things get to me more then I should. But we’re still not having sex. It makes me insecure because I no longer think he’s attracted to me or loves me anymore. I know sex isn’t everything but it just doesn’t seem like anything is there. There’s no way he can be cheating on me because if he’s not at work he’s with me literally at all times. He calls me immature because I get upset over it and sometimes it causes a fight. But he keeps telling me “sex isn’t everything” “I’m just more grown up then you to know there’s more to life then just sex” which I understand that but at the same time it doesn’t take my insecurities away. Should I be worried? And what should I do?