Son had 104.1 temp

My 6 month old son woke up this morning with a temp of 104.1. He’s had a small fever since Tuesday, Monday he had his 6 month shots so I thought the fever was from that. He had rsv roughly a week or so ago and the rattling sound and coughing came back today but not as bad as when he had rsv. Obviously I took him to the ER they swabbed for the flu and did a chest x ray. Flu was negative and x ray came back good. They said it was just a virus and told us to switch Tylenol and Motrin. His temp is now 102.4. What I’m wondering is if I should take him to the Children’s hospital? His temp went up 2 degrees in 40 minutes.

My husband and I lost a baby in February 2018 due to a bad heart defect, so when one of our babies/kids get sick we get a little paranoid.