Coughing and cold

Johanna • First time mommy to Max, born September 5th 2016 ❤️
My little dude has started coughing a bit yesterday. By a bit I mean very brief cough and less than five times a day. Now (today) it got slightly more hoarse and frequent and it's clear that he has some mucusy stuff in there. Can't see it but I can "hear it" (sorry if this sounds weird). He also sneezes fairly often and seems generally congested. 
He has a 99.8 temp and he's still eating ok. It seems to be a common cold. But I've never heard him cough before and as a new mom that kinda worries me. Would you go to the doctor for this? 
I'm sort of afraid to go because last time we went for a checkup at two months we were both healthy and we both came back with a horrible cold from sitting in the waiting room...