4 weeks pregnant

okay so I'm 4 weeks & 5 days today. It's my second pregnancy, (my first born was born at 23 weeks & only lived for 5 months). I'm a mail handler at USPS & am constantly lifting and pushing around heavy things. Do I need to go on light duty now? is this pregnancy considered high risk because of my last pregnancy. I asked my doctor, she said I have to go on light duty when I hit 4 months but I feel so guilty lifting and pushing heavy things constantly at my job, please, I need some opinions!