Weight loss while pregant.

I'm currently 10wks and 4 days. In this time frame I have lost 11 pounds and it seems to keep coming off. I'm not really complaining considering I have been struggling to lose weight for months however, this can't be good right?. I know weight lose is kinda normal during the first trimester especially for a woman of my size but, how much is too much? When should I worry? My doctor's know about it but haven't made mention of it. Can it harm the baby?