Fever in baby?

Jessica • My rainbow baby was born on 7/25! 1 MC in May of 2014. PCOS sufferer, conceived without medication, I used plexus!! :)
Around 2 hours ago I took my 3 week olds temperature because he felt warm to me. It read 99.4 degrees Fahrenheit. I've continued to monitor him and now it's down to 98.6. I'm going to keep checking and call the pediatrician in the morning just to get a professional opinion. I wouldn't be so alert about a low grade fever but my son has a heart condition, and they said any illness can be worse for him than a baby without the condition.
Just asking for opinions if you've seen your babies temperature fluctuate like this?