In love but...

I love my s/o but at the same time he annoys me. I want to be with him yet I don't know if I do. But the thought of leaving scares me.