Gallstones at 5 weeks pregnant

Doctor says I have gallstones and I'm  5 weeks pregnant but they not inflammed should I worry