Low milk supply at night?

My son is 5 weeks old and we’ve had no problem until this week. In fact he was gaining a pound a week for the first 3 weeks. Anyway, in the middle of the night/early morning my breasts are so full that it hurts and I’m leaking everywhere and I hand express a little to relieve the pain if he’s not awake to eat. My supply seems fine after that until about mid afternoon when suddenly I don’t even have a let down (compared to morning when I have multiple let downs per session that are very rapid). He sits there sucking but not swallowing and eventually gets frustrated and starts fussing or screaming. I feed him on demand and I don’t supplement because I don’t want to make the problem worse. I just don’t know what to do he’s been nursing on and off for the last 2 hours and I haven’t had a single let down and he refuses the pacifier. He bangs his hands on me and starts crying And kicking his feet. Either latches on/off and fusses or he latches extremely hard and goes crazy like he’s trying to get more out. Any advice?