When should I be worried?!

I am breastfeeding my 5 week old. I have on two occasions gave her enfamil NB 1-2oz only as a supplement when I wouldn't have enough breast milk. However, after thinking she had a milk intolerance, I decided to try enfamil alimentum all of Tuesday (was going to give it a try for 1 week and see if it made a difference in her behavior/tummy issues). I noticed she was even fussier that day and only pooped once that night. It is now Thursday and that was the last time she had s dirty diaper :/ after seeing that she was fussier that Tuesday night, I stopped the formula and just Breastfed her again and have been ever since. When should i be concerned and take her to the dr!? She used to have up to 3-4 poop diapers a day before this.