Refusing bottle after 1oz

My LO is almost 12 weeks old. He is EBF but we introduced a bottle at about 4 weeks for when we were out and about. He always did great and drank the whole thing. At 10 weeks he started refusing the bottle, and only wanting to nurse. He will take 1oz fine, or with a little fight, but then he gets really mad and refuses to eat more until it's been another 2 hours. I returned to work part time last Monday and he pretty much waited to eat all day. He did 1 oz for the first 3 feedings, then 3.5 oz from the bottle right before I came home. (He usually only eats about 2.5 every 2 hours.) I hate to think he is starving himself all day waiting for me to be home to nurse. Has anyone else had this problem?