Sick preemie

Edit**** I ended up taking him to the ER for a fever and his cough. Turns out he got pneumonia. ****

I have a 6 month old who was a preemie. Hes adjusted to 4 months. He has had a bad cold for 5 days. I took him to the doctor and she said he looks good just has a cold and turn the shower on. At that point he was sick but still kinda happy. The day after the doctors he started crying ALOT and eating maybe 1-2 ounces at a time.. (normally he eats 6-8oz) he sleeps 20-30 minutes before hes up crying inconsolably. This repeats all day and all night for 4 days so far. He barely even opens his eyes just crys or lays there and whimpers til he falls back asleep.. he has had a fever that stays around 99.9-102 tried everything. He so congested it sounds like he is snoring all the time. Humidifier, hot shower, nasal spray, suctioning, baby chest rub, cool baths. When he cries I change the diaper, try to feed him, rock him, play with him, just holding him, nothing is working. We all had this same cold but none of us had it like the baby does....

Now my question is should I go back to his doctor or a hospital? (i feel like the doctor just dont care or listen to my concerns they always say hes fine).

Is there anything else I can do to help his bad cold that seems like it's not going away or lightening up?.