Newborn baby fever: go to hospital?

Our baby has a fever of 101.0 (armpit), or 101.6 (rectal). Do we go to the emergency room? We are getting conflicting advice from our two doctors.