SO Doesn't Want Me to Breastfeed...

My SO told me the other day he'd rather me not breastfeed. The reason is because I do have depression and suffer from terrible anxiety, and I haven't been able to take the Ativan since I became pregnant. It's been a very tough pregnancy for me so far, and he thinks I should get back on the Ativan asap after the baby is delivered. I'm a FTM but apparently I could still pass the drug while breastfeeding?

Anyway, I'm really looking forward to breastfeeding and I'm not sure I want to give that up. He had a valid point that I do better than the Ativan than without, and there is some fear that i might experience postpartum depression. I guess I'm just sad to think I won't have that special time to bond with baby during breastfeeding. Thoughts? Not sure what to do....