Mild mastitis?

So on the 19th the outside of my right breast started to feel really sore, and at first I thought it was just from working out cuz my entire body was extremely sore. After I worked out again I took a shower, nursed my baby, then a little while later i nursed him again, and I suddenly got the chills really bad, along with body aches and pains, headache, and a mild fever. I felt like crap!! The next day it wasn't quite as sore, but the area where it hurt turned red and warm. Now today, I feel pretty good and my breast isn't nearly as sore as it was, but it's still a little red. It never did hurt to nurse. All of my symptoms pointed to mastitis, but it wasn't nearly as painful as what I was reading it should be. Is there such a thing as a mild case of mastitis? Not sure if I should go to the doctor or just continue to try and nurse and pump as much as possible on that side? Has this happened to anyone else and didn't have to go to the doctor??