Not eating more than a few minutes at breast

Kylie
We introduced the bottle several weeks ago because we knew that last week hubby was going to have to watch our LO for the day. We've also had family watching him for a few hours while visiting them so we could have a few dates nights since family is so far away. 
Over the past two days, LO has hardly been eating at the breast but will consistently take 4oz by bottle. Basically he is eating while there is a let down bc then he doesn't have to work for it, but after that he doesn't want to eat. 
During the night he'll take the breast no problem which is good bc we can be back asleep within 20-30 minutes but during the day he's not. 
Any recommendations other than eliminating the bottle to get him to eat at the breast past the let down. If I try and keep him at the breast he just spits it out and then cries.