Had to give up breastfeeding

So I wish I could say that the doctor told me to switch to formula but no. So 3 months before my due date I found out the dad was cheating so I left and moved in with my mom. Well there was a week straight where baby was up all night clusterfeeding and I got literally no sleep for days so I would have coffee to get me through the day. She thought that was why baby wasn't sleeping so for weeks if I had so much as a sip of anything caffeinated (even soda or tea) I would get all but attacked and made to feel horribly guilty for needing a pick me up. So after weeks of the stress and guilt I tried to cut out caffeine, she didn't stop and when I just couldn't do it anymore I just said screw it and switched to formula just to get her off my back. Well she stopped about the caffeine...and now its about how im always holding him. Im pretty sure he misses the breast and just needs that comfort so he cries literally any time I put him down. So now im guilty and honestly depressed over not breastfeeding him and im beginning to resent her completely. And just an FYI the doctor even agrees with me that its not the caffeine but apparently she's smarter than a doctor 🙄