Night feedings

I need some advice!! My baby is 3 months old but was born prematurely 2 months. At night when I try to feed him he goes crazy will eat for a couple mins and then just cry, I try and burp him then try and get him to latch again but he gets so upset that all I can do is lay him on my chest to calm down. I know he should be hungry cause it's usually been 5 to 5 1/2 hrs since he ate last and he will suck on his fists. What am I doing wrong ? I feel so bad for my baby to cry like that knowing he's hungry but won't eat. I need some advice I'm about to pull my hair out