How do I handle this?

So my boyfriend has rejected my sexual advances lately. His most recent one was when I went to touch him, down there, and he said to me "Having a boner is uncomfortable" then pushed my hand away.

Now I'm to the point where I dont even want to have sex. I am tired of having sex only when he wants to, and now I'm being rejected. How should I handle this? We recently had a serious conversation about some other things and I dont want to come off as a nag, so I'm not trying to have a serious talk about this right now with him. Should I just reject him if he tries to make a move on me next, or tell him the way he has handled things has turned me off to sex? I have never dealt with this before so I honestly dont know haha Thanks for the help!