Baby won't eat!

So my baby will be 6 months January 19th and I'm having a really hard time feeding him. Lately he's been getting up about 2-3 times a night and eats fine but once morning comes all hell breaks loose. For the past few days he cries to eat about every hour but when I try to breastfeed him he'll only eat for a couple of minutes before either pulling off to scream or to look around the room. I tried giving him bottles but he'll MAYBE take an oz and then screams with those too. Tonight I gave him his oatmeal and he devoured it so I know he's hungry. He's pooping and peeing but I'm worried because before all of this happened at his last drs apt he actually dropped on his growth curve and the dr seemed concerned. Would you call the dr to let him know or wait it out? Thank you!