Getting frustrated

My boyfriend and I have been together 3 years and live together. Plain and simple- I try to be spontaneous and keep things interesting, and I love going down on him. So at random times I always try to, to surprise him and make him happy. But instead he is always pushing me away and saying no no, not now. I mean come on, what guy would turn down a BJ 90% of the time?! We've had issues in the past with his infidelity, and I just want to feel like I can satisfy him. Of course that's not the only reason, I want to do it too. But it's starting to really upset me and make me feel insecure and awkward. It really ruins the whole mood of everything. I don't know what to do. And he told me that he absolutely loves when I go down on him and that I am the best he's ever had, so it's not like he doesn't like it. He just will not let me do it. He used to love when I did it, and would even initiate it himself like, for instance, if we were at a party, he would sometimes take me into a hidden room or spot outside and push my head down (playfully of course) and I would be excited and totally enjoy this kind of stuff. But now it's just boring and annoying and sometimes hurtful, especially for someone who already has insecurity issues. Our relationship is fine otherwise. But I'm worried there may be some underlying problem or secret. Oh, and we're not really having sex that much either, which is probably obvious. Any advice or experience? Thanks!