Am I being too paranoid?

I have a one year old daughter who has been fighting a fever since Thursday night 🙁 She had a fever of 102 that night so I gave her tylenol and gave her a lukewarm bath and put her to bed. On friday her fever was up and down from 99.7°F to 101.7°F so I called her doctor and asked if could take her in to get checked out and he said "It's most likely a viral infection. If her fever is below 102°F and she's still eating and drinking fluids and having wet diapers then she should be fine just give her tylenol, keep her hydrated, and let her body fight off the infection." She's still eating (not as much as she usually does), and drinking lots of water, shes being a bit fussy. The only other symptoms she has is that shes really gassy. I want to take her to urgent care right now but my SO thinks I'm overreacting and that they're just gonna tell me the same thing her doctor did. I'm worried because its now day 3 and I gave her doctor another call and he was so rude and told me the same thing and hanged up on me! I'm her mother I have a right to be worried! I also got in an argument with my SO because he said "You can take her if you want but i don't know how you're getting there. Im not wasting my time or gas when I know exactly what they'll tell you. I'm going to bed you're being so paranoid." I just don't like seeing my baby not be her normal goofy sweet self. ❤ Should I just trust her doctor or should I be the paranoid mother he said I am and take her in?