fever in 11 month old

Hannah
My son is 11 months. Yesterday his daycare called me because he had a fever of 102(oral) I took him to the doctor and he said to give him Tylenol. Overnight his fever reached 103 (rectal) but broke at 7am. Tonight his fever is back at 103.5(rectal). My doctor keeps saying to give him Tylenol. At  what point should I take him back to the doctor or to the hospital? This is his first real fever and I'm super concerned. TIA.