Am I Wrong For Saying No?

I've been debating on posting this. But I finally decided that I needed some opinions. So, I've been with my S/O for almost 2years. And I've had Mirena for 3years. The Mirena has lowered my sex drive. I rarely ever get horny. And I have a harder time getting wet. So it has made me a bit depressed. Anyways, there have been times where my S/O has wanted sex and I have said no and he has respected that. But there have been a little more times where he just makes a face when I said no and says, "I promise it'll just be quick. Please?" And he won't really take no for an answer. He'll keep pushing it. I usually just give in. And I've told him that I don't want sex right now and I keep saying no, but when we do, I feel horrible cause he hits the right spot and I do enjoy having sex a little bit. But like I don't know what to do anymore when he gets pushy. Because when I've stood my ground, he gets upset and I feel like a HUGE jerk. So am I wrong for saying no?