I think I should switch to formula

I don’t get it my baby is 5 weeks old and does so well during the day will sleep good between feeds and eats well. 6 o clock every evening baby will get very upset crying all the time unless she is nursing. She drinks me dry and then gets mad cause not a lot of milk is coming and she will do this until 11 pm of frantically nursing and crying then she will sleep for 4 hours+ it is very stressful maybe I don’t have enough milk , my boobs feel empty this last week I bought formula tonight and think I will switch tomorrow ... :-( I love breastfeeding but my baby is starving every evening. tonight I am just watching her sleep I am crying I feel like a big failure.