Fever or no

My baby is 4 weeks old today. He has congestion in his nose, however eats and sleeps as perfect as usually.

Used some saline drops and the bulb and got out mucus.

Have a humidifier running.

However he had a 99.5 rectal temp this morning. I thought maybe it was his clothing and blankets since his head and feet felt fine so I removed them checked again 20 minutes or So later so I checked again and it was 98.6. Few hours later went up to 99.7 again but now he’s napping.

I called the dr of course and they said anything 100.4 and over is more worrisome. do you guys agree with this as well? Sorry paranoid momma here.