Bipolar 2 and pregnant

I’m 19 weeks and I have been struggling this entire pregnancy. I was in lamictal for my bipolar 2 for a year and a half and quit as soon as I found out I was pregnant because I was worried about side effects and harming the baby. I’ve talked to multiple doctors and they say I can go back on my medication if I need to but I am so paranoid something bad will happen or if there was a birth defect, I would blame myself completely so I’ve said no but I am struggling so bad and I’m only half way through. Anyone else on medication for mental health? If so, how’s it going?