Should I take LO to the ER or not?

I have a 10 day old. He was very very fussy last night though this morning/afternoon and cried every time I laid him down so I held him the whole time. Now he is sleeping but would have slept through his past 3 feedings if I hadn't woke him, and he is dozing off 10-15 minutes in. He is flushed red but his temp is just 99.2. Should I take him to the ER? I'm so scared to expose him to all those germs for nothing but don't want to ignore this if it's bad.