When to take baby in for fever?


We went to the doctor this morning all tests were negative. He prescribed Tylenol and an antibiotic to fight off any future infection. All day his fever has been between 103.8 to 104.5. He's 10 months old and I can't get him to eat much or drink much. I've given him baths put a wet towel on his forehead but for some reason I can't get his fever to drop below 103.8. I'm waiting for a call back from the doctor but seeing my baby in so much pain has me so worried and wondering if I should take him in to the ER? Any advice mommas?