Breastfeeding question - want to start after only pumping

My son is 5 weeks old today. Up until this last week I have been exclusively pumping since there was a concern about his weight gain during the first week (I liked to see that he drank enough ounces after that). Well I have really wanted to breast feed and he latches very well the only problem is he pulls off after a couple gulps every time and he ends up getting frustrated. I think it's because my milk comes out too quickly and he can't keep up. Does this sound possible? If so what can I do? I really want to have that closeness with him and I'm getting sick of getting up in the middle of the night to pump after bottle feeding him:(