My baby is too tired to nurse :/

My little one (3w5d) has to be supplemented per his pedi since yesterday. I gave him two 2oz formula bottles yesterday, and he's had one today so far. Since giving him the formula, he just falls asleep when offered the breast. Should I just supplement him today to get his calories up so he's not so tired, and pump when I give him the bottle? He's peed 4 times today, but no poopy diapers. He was getting enough breast milk, but wasn't getting the fatty milk he needed to gain weight, which is why we are having to supplement. Thank you guys.