Is this normal during breastfeeding?

Hi ladies! So I’m on the second month of breastfeeding 7.5wks to be exact and I think my supply is winding down… My baby is super sleepy and after a few min on the boob falls asleep.. I try to make him uncomfortable by changing his diaper etc but I’ve noticed that by the last nightly feeding around 10pm he’s pulling on the nipple and getting fussy. I use the Hakka pump so I’ve been giving him 2oz or so that’s collected during the day and he devours it! My breasts are super sore and I’m freaking out is my supply running out or is this just a stage? Thank you!! I don’t pump btw.