I feel so guilty

I feel guilty when I get frustrated with my baby when he won't stop crying. I love my baby so much but sometimes I just get annoyed and want to cry because I hate seeing him like that and I sometimes can't get him to stop. Anyone else?