breastfeeding issues

i really need advice. i have a bf / father of my kid who used to be very supportive of me breastfeeding. the last appointment we had at the pediatrician he came with me. normally he works during the day so i have to bring my son myself. last time we went, the doctor was only slightly concerned because our son was along the line of underweight (keep in mind he was born two weeks early). she told me to make sure i pump more so i can increase milk supply and get a sort of storage up. i’ve been trying my best so he can gain a lot of weight before his next appointment. but, my bf says every single day when he comes home from work that i’m not feeding him enough. my son is a clingy baby so he really only likes to mainly sleep on me. so without asking me nor the doctor he has been feeding my son formula. he just expects me to not care even though i’ve been trying so hard to make sure i keep up. he says i need to not put my ego before our son’s needs. but, if our son was starving like he says he is then why wouldn’t our pediatrician say something more during our last appointment? she didn’t even mention formula? wouldn’t she tell me to use some formula? im so conflicted. my bf who once was extremely supportive now thinks i can’t naturally feed our child. i also just called a lactation consultant from WIC since i’m so concerned. she said i just need to nurse more and it will increase my milk supply. i don’t know what to do now. help?!?