Ladies I need advice 😰😰 plz help!!

I work for the post office at the plant. So it’s a factory setting such as lifting 60-70lb sacks over my head and pulling pallet jack with big watermelon boxes full of mail... with that being said I have turned in paper work to be in light duty due to my pregnancy I am currently 17 weeks pregnant. I explained to my doctor how I had been feeling and my troubles at work and he suggested I go on light duty. The paper work my nurse gave me the nurse had wrote down “she feels she can not keep up at a fast pace and needs to be on limited to duty,” I felt wich kinda threw me under the bus... that being said my jobs not willing to accept this due to the wording. After I turned my paper work to my supervisor, he just kind of looked at me and said u can’t be in light duty just because “you” think you can’t keep up this is not except able..... then lot of other employees found out and began stating “oh I worked up until I was ready to give birth” “I did it all with no limitations” just making me feel really bad about myself..... my question is is it silly for me to be this upset or on light duty this early. I need advice 😓😓