What to do? Breast supplement, I need advice!

Reb • 👦🏼💚👼🏻🤰🏼💙
My son is now 5 days old. The doctor told me just yesterday to start supplementing because he lost more than a pound from birth, just to get his weight back up. She said to nurse for 10-15 minutes and then give formula. Well, I've been through 3 feedings with him and he refuses to latch to me. But will to the bottle. He took a little bit to latch to me before but would eventually. Should I start pumping and giving him breast milk through a bottle or keep trying for him to latch and don't give in??? I wonder if my nipple isn't in far enough to stimulate his latch.