Baby refuses to re-take the bottle

Was wondering if anyone had a similar issue with baby not retaking the bottle or if anyone can help. When we’re bottle feeding our baby, he will drink until we take the bottle out of his mouth first time, then when we put the same bottle back in his mouth, he refuses to take it, and we know he’s still hungry. We’ve tried different types of bottles, and he does it with both breast milk and formula. We’ve tried everything, but just can’t figure out why he does that.