Hearing concerns

My baby boy will be 4 weeks old on Monday. He was delivered 3 weeks and a day early at 36w6d via emergency c-section because he was breech and my labor happened very, very quickly. While in the hospital, everything went well as far as his testing, Apgar scores, reflexes, etc. When they did his hearing screening, his right ear passed but the left ear failed. They immediately retested the left ear when I got upset (FTM!) and he passed right away. However, I am noticing now that we're home, he's not really responding to sounds. There have been occasions when he's startled upon hearing loud noises or responded to a certain song I play on my phone, but it's not consistent and may just be flukes. We have a dog who barks constantly and he doesn't even flinch. I am terrified that something is wrong. I mentioned it to the two pediatricians we've seen; one said not to worry, but to keep an eye on it and to talk at his one month appointment and the other said it's not atypical and not to worry. The woman who screened him said sometimes c section babies have fluid that doesn't get pushed out as it does in a vaginal delivery and that it's not uncommon for early and/or c section babies to fail. She didn't see a reason to return the next day and test him again so he was considered "passed" in both ears. The one pediatrician did say he'd had patients who passed and did end up having hearing loss later though. I am so afraid that I did something during my pregnancy that caused this and that something is wrong with his hearing. I have Googled til my eyes crossed; does anyone have any experience with this or suggestions for what I should be doing? I would really appreciate any responses.