Brick dust? I’m about to give up ☹️

My baby is 4 days old, we have his follow up with his pediatrician on Wednesday. He latches well, I hear the ‘suck suck swallow’, he stays on my breast until I feel it soften out and then some sometimes. He feeds every 2-3 hours and on demand whenever I catch his cues. I only have one functioning breast (due to surgeries on the other.) I can feel my milk transitioning from colostrum and coming in (tingling, heavy, sometimes a little hard) but I haven’t gotten ‘engorged’.

Baby has pooped 10 times since birth, he’s produced the correct amount of wet diapers for his days of life, with the exception of no poop today. But each time he pees there is a bit of brick dust in the diaper. I’ve read a few articles and theyre making me extremely nervous as it says usually brick dust should disappear or be gone by day 3, and here we are going into day 5.. is this a sign that I need to supplement him with formula? Or do I just give it another day or so until we see the ped on Wednesday? I wanted to badly to breastfeed him and I’m trying my hardest, we don’t skip any night feeds, I wake him to feed when hes napping, I make sure he has a deep correct latch.

I feel like I’m hurting my baby instead of helping him.