Quitting breastfeeding?! HELP!!!

My son will be 3 weeks tomorrow and we made it about a week of exclusively breastfeeding but it just wasn’t going smoothly so I started adding in the occasional bottle of pumped milk to get a break from him being attached to my boobs. I soon began just pumping exclusively because we just weren’t clicking together with breastfeeding, it was more stressful than anything on BOTH of us. Now, I find myself constantly feeling like I’m either pumping, feeding the baby, washing pump parts, or sitting there rocking my awake baby thinking about the fact that it’s time for me to pump but I can’t because I have to hold him. I have been finding myself crying when he’s awake and I need to pump and I’m afraid it’s going to lead to depression. SOOOO, all that being said, I’m highly considering just switching to formula. I’m having a hard time with it though, feeling guilty and like I’m letting my baby down and that this is somehow going to make me seem incompetent. Any advice or stories from anyone who has gone through something similar would be greatly appreciated!!!!!