3 week old only eating a few minutes

My son is 3 and a half weeks old. He has a lot of wet and dirty diapers, so I'm not super worried but I'm having trouble getting him to eat more than 3-5 minutes on either breast. He is also getting super fussy at night and constantly pulling off my breast and then relatching and crying, but he rarely cries during the day at all. I'm just wondering if that's normal or anyone else knows why he's doing this or if there's anything I can do to help him. I'm wondering if it's a growth spurt or something.