I'm 32 weeks + 5 days, currently loosing weight

I've had a tough pregnancy and lost 7 lbs my first trimester.. I gained the weight back plus 4 lbs. I was almost 100 before pregnancy, dropped to 92, then finally I was slowly growing back up to 107. BUT the last week, I've slowly been dropping and I'm 104 from the last 4 days. I've been on a no sugar diet so it's been super hard, not eating as much because of my options. No sugar diet is due to yeast. I don't respond to western medication so my midwife cut me off of sugar and put me on prodiotics and a whole bunch of natural remedies. Should I be concerned?