Breastfeeding concern

My son does a really good job nursing through the whole day. Than at night he gets super fussy. He is hungry and wants to nurse but he latches for a little bit than forcefully pulls himself off. When I try to and have him latch again he won’t latch and cry’s and keeps pushing me away even though he is hungry. He will continue to latch and than let go and this go on and he gets more and more frustrated. This has happened 3 nights in a row I’m not sure what I’m supposed to do.