Blow jobs

Ok so me and my boyfriend have been together for about 2 years and i have given him blow jobs before but it seems that whenever i disagree to give him one he gets mad and won't talk to me for an hour or so, till he starts acting like nothing happened. Is this normal or should i talk to him about it?