Struggling with breastfeeding and formula

My baby is 3 weeks old and i had c section and my milk came after 3 4 days now he is on formula and i am trying my best to get him off it and exclusively breast feed but his he hungry after every feed so i have to give him formula my supply is also less but i want him to breast feed only if anyone have tips please share…..i need to get rid of this formula….i try to feed on demand too but by taking formula he sleeps for 4 5 hours…1st time mom bit anxious..🥹