When is a lack of sex a problem?

I need some guidance, my boyfriend and I aren't having sex. He warned me in the beginning that he had some sexual trauma from his childhood and it sometimes affects him still. I asked as many questions as I could about how it affected him, how badly it affected him and if there was anything I could do to help. He assured me that from his experience his trauma would affect his ability to get it up every once in a while but if it was an issue he'd make sure I was taken care of. I figured that wouldn't be an issue so I went ahead and got in the relationship with him, I understand his issues and that I'd have to be patient with him. Fast forward 7 months later and we've had sex all of 3 times, we've attempted a lot more than that but he just can't get it up. Every time we run into that problem I'm supportive, I tell him its okay I'm still here and we move on but he never even attempts to help me out. He gets emotional and is in a mopey mood the rest of the night. I feel like I'm walking on eggshells because any time I bring up something in our relationship that bugs me he throws himself into a panic attack and swears that he's the worst person ever, im gonna leave him, and then I worry that he's gonna hurt himself. I've tried to talk to him about solutions, like therapy, vitamins, medication, literally even masturbating and he has an excuse against everything. I've been patient and understanding  but its feeling useless at this point because he won't take any steps to help his problem. I'm at the point now where I feel like I'm in a relationship with high school level intimacy and no support. I can't communicate what I feel and I don't know what to do. I've loved him for a long time and I want so desperately to make it work but I need more.

What do I do?