Need help

Baby is 5 weeks. He is constantly acting hungry for the past 3 days. Figured it was a growth spurt again but I'm not sure anymore because he isn't wanting to nurse. He is crying constantly and keeps acting hungry (bringing hands and sucking, turning head to breast) but when I go to nurse him he eats for a minute or two and then begins to get upset. I know he's getting milk because he still has some in his mouth when he pulls off and I can hand express a good amount too.. I just don't know what to do. Any suggestions?