Pouty S/O

My boyfriend and I have been together 4 years. Ever since we've moved in together, my sex drive has been kind of low- for whatever reason, tired, don't feel like it, whatever. Anyways, last night he wanted oral sex and I said no & he got mad at me and just rolled over and pouted. This happens kind of regularly and it just makes me mad that he gets mad! Any advice?