My baby is sooo sick😞

My poor little lady is burning up ...she’s had a low grade fever for a few days, but nothing too bad. Last night she was upset while trick or treating, but nothing too terrible. She fell asleep in my arms, and when I went to lay her down realized she was soooo hot. I took a temporal temp, and it was 102.9. So I did a rectal for more accuracy, and it was 103.6! I freaked and called her doc, but they said she should be fine. She’s been miserable and burning up all day, but it hasn’t gotten over 104, which is when they said to bring her in. Has anyone else dealt with this, and at what point did you bring your little one in?