Breastmilk and Formula Trouble

I've noticed that whenever I do just formula feedings, my baby will keep a pretty consistent schedule and eat the same amount. Eat a lot and sleep a lot. But when I switch to breastmilk for one feeding, it throws off the rest of the day even when it goes back to formula. Eat a little and sleep a little (making him cranky too). Has anyone else experienced this as well? I'd like to continue pumping breastmilk, but it throws our day off.