Very small 5 month old

At my sons 4 month appointment he weighed 12 pounds 13 oz and they said that was pretty small and wanted me to start giving him bottles. He normally only breastfeeds for maybe 5 minutes. But he will not take a bottle. Now he will be 5 months on the 9th and it's still the same. I wanted to take him in today to do a weight check because I'm starting to worry. His 3 month sleeper is looking a little loose on him right now. Has anyone been through this? I can't get into the doctor until Wednesday. Thank you!