Need some encouragement

My baby is 3 weeks old and breastfeeding has had its ups and downs. Some days go by fine and I feel like I can do this no problem, but other days are a struggle... I’ve dealt with the cluster feeding but last night it seemed like no matter how much I put him on the boob he was still hungry. I mean, screaming and fussing until 5 in the morning when I finally gave in and gave him some formula. He’s gaining weight fine and has plenty of wet/poopy diapers so I’m sure he’s getting enough... but I know giving formula can dwindle your supply and I’m already paranoid enough about not making enough. Is it really that sensitive that one bottle could ruin my supply, or on nights like last night and to maintain my sanity, can I give him a bottle?