Fever from the flu at 14 weeks

Can I please get some reassurance. I work at a hospital and they had a influenza outbreak at one of the wards. I thought i was far away from there but i guess somehow maybe a staff brought something back or i caught something at the cafeteria...2 days later i had everything from the flu. Fever, chills, aches...i've seen my doctor and when i was there my fever was only 37.7 (99.9) but by the time i got home it went up to 38.8 (101.8) and it's been bouncing around from 100-101.5 for the last 2 nights. I'm really worried the fever is going to harm the baby. When i asked my doctor if i should take tylenol he said it wasnt high enough to take medication and that i should go to the hospital if my temp reaches 40°.

I dont mind the gross cough, the stuffed nose or the sore throat..im just worried the fever will harm my baby. Has anyone had experiences with fever when carrying?