Gallbladder removal | Gallstones?!!

Had my daughter November 2020. About 3 months after having her I started having horrible attacks, radiating through my back & making it hard to breathe. I have had about 4 total up until today in February 22'. Went to hospital & they found multiple gallstones, 3cm in size & sludge. I had a consultation with doctor & he said to remove it. Well, I went ahead and stuck to a low fat diet & he called," If symptoms have gone away, I don't recommend surgery if you can treat it with diet. Call if you need to set up surgery down the road." That's what I've been doing about 3 weeks, but symptoms are coming back.. Now they're in my left side too where as before it was mainly my right. I'm worried about pancreatitis now but he had said it doesn't sound like that. It's just weird how it started out of nowhere. Me & my husband want to try for a baby in May/June but I read up gallstones can cause problems. I just feel not like myself getting an organ removed at 23... I need opinions, advice, anything. I understand this is Glow & I should go to a doctor but I am specifically looking for other people's experiences. Thank you!!!♡♡♡