Marriage advice

I'm just one month married. On our anniversary which was Tuesday, in the midst of a conversation, my husband confesses that sometimes he thinks that I will run off and cheat on him since he doesn't give me enough time physically or sexually. Of course he admits to his fault on this but clearly doesn't do anything to resolve this issue. I got angry because in the midst of this comment I felt as if he just belittled me and put me down as a potential adultery or some slut off the street that can't contain her urges. Since Tuesday, I have been upset and haven't slept in the same bed with him. He says I'm taking his comment to seriously and that this whole thing is more in my head and that I'm just trying to create an argument. I have been with him for 9 years off and on and have never once cheated on him or have him reason to believe that I will. I want to fix this but he's offering no apology on what he said and the way he made me feel. Should I just let it go or should I get him to see my point of view and resolve this?and if so, how should I approach it?