advice for FTM

So I gave birth to my son 1/14/2017 at 4 am. He weight 6lbs 6oz at birth. I did skin to skin as soon as possible but he didn't latch, not to mention I didn't have ANYTHING at the breast at all (even today 1/17/17 I just have minimal colostrum) but since then while in the hospital I had attempted to nurse to no avail. Eventually I was so desperate to feed my child I just supplemented with formula. He was extremely slow to respond to eating... 5-10 ml at a time every 3 ish hours on the first day. Now to day 3 he is still difficult to feed- getting 30-45 ml every 2-3 hours. At his first weigh in yesterday he weighed 6lbs 0.8oz. Doctor stated he needed to be eating at least 2 oz at a time every 2-3 hours to gain weight back but he fights the bottle and I cant force him to drink. I do as much skin to skin as possible and I'm still pumping and putting him to the breast to hopefully stimulate my supply. Anyone have any experiences like this? I am extremely worried.