Pregnant with pcos

My new OB is trying to take me off of my metformin and I don’t feel comfortable with his decision. I’m 14w2d( second trimester) and have been taking it for 2 years now. When I was working with my fertility specialist they wanted to keep me on it throughout my entire pregnancy and now my ob wants to take me off because I wasn’t actually diagnosed with diabetes. I really want to trust his professional opinion but it’s hard especially after doing my research and 2 MC in the past. I’m worried my doc is saying since I’m out of the 1st trimester I should be fine. Have any of you ladies experienced anything like this, if so how did you handle it?