When to take baby to ER???

My LO has not felt good for the past 4 days now. Wednesday she refused all bottles after 3pm and was extremely fussy and had 3 very loose stools. Wednesday night she tossed and turned all night and developed a slight cough. Thursday morning she took one bottle and her cough was get a little more regular. Thursday we called the doctor and they told us to bring her in so we did and they said it was an upper respiratory viral infection so all we could do was treat the symptoms, but she told us to give her 12 oz of pedialyte to help prevent dehydration. Thursday all she had was 6 oz of pedialyte and 3 oz of formula. And then had diarrhea. Friday she started doing a little better she had 12 oz of pedialyte and 12 oz of formula. But her cough sounded worse and she developed a runny nose and had 4 diapers full of diarrhea. Today she is refusing to eat again. She's has 3oz of formula and 3 oz of pedialyte. Her soft spot seems sunken in and she had another diaper of diarrhea. She did have a wet diaper today though too. No fever but cough and runny nose are even worse and all she does is cry and she's normally a happy baby. 
Anyways at what point would it exceed your comfort zone where you would take your baby in? I'm a FTM and I don't know what to do?!? I keep trying to call the doctor but all I'm getting is their answering service....