Scared of boyfriend

I’m unhappy in my relationship with my boyfriend whom I live with. I don’t feel wanted anymore, he doesn’t really have the desire to have sex anymore which makes me feel bad about myself. He says it’s not me and that he’s very attracted to me and he loves me. He makes me feel shitty about myself sometimes because he says he doesn’t like it when I drink (because we fight a lot when we drink) and he makes it seem like I’m such an abusive person when I drink. But I’ve NEVER been an angry drunk, EVER. He gets super cocky, ignores me, and makes comments that make me feel looked down on when he drinks so he gets me upset in turn. But he makes it seem like I can’t drink without getting angry and gets mad at me when I even sip on a glass of wine. The first time we drank and got into an argument while drinking I was continuously telling him to get out of my room(bc we have separate rooms but always sleep together) but he was blocking the doorway and was trying to push his way in so I tried to close the door and tried to just push him enough to get him out of the doorway and he shoved me across the entire room and I fell back and hit my head on the dresser and I got super scared and called the cops because we had just moved in together in his hometown and he was the only person I knew and I freaked out. He cried and said he’d never hurt a girl before, and he seemed sincere and I forgave him. Since then he’s gotten so pissed that he’s punched a hole in our hallway and in our bathroom door during other fights. Am I wrong to be a little scared? Because even though “at least he’s hitting a wall and not me” that anger is directed at me, meaning he wants to punch me when he does that. And what if one day he can’t control it and it’s not the wall? I really do love him and he doesn’t seem like the abusive type but I can’t help be worried. And I’m having second thoughts because of that and the fact I feel undesired. I also don’t know what I’d do if we broke up because I don’t know if I can support myself