Starting formula¿

I’m almost 6 weeks in with my son breastfeeding and we introduced formula probably way too early since I wanted to continue breastfeeding but it’s just getting really hard to feed him and he’s not getting enough with me anymore I feel like I should keep trying but I also have a feeling I’m not feeding him enough either. He latched on better than he did before but he’s letting go 2-3 minutes in as if it’s not coming out anymore and so I switch him off the other side and he does the same..

Would it be bad to just do formula now on? My sister, when I was still pregnant, kept calling me selfish and said I’d be a bad mom if I didn’t breastfeed and my mother keeps telling me to do bottles so I’m really confused and conflicted between both parties but it’s really taking a toll on me to co to us at the same time :(