Could my baby be allergic to almond milk ?!

Every few mornings I will have cereal but with almond milk because my baby reacts horribly to dairy😳 well right after I eat cereal with the almond milk he cries and cries and cries at my breast he’ll get on and off and on and off and scream and cry all day until night time (until it’s out of our systems I guess?). Then the mornings I don’t drink it he’s perfectly fine all day?? Does this pretty much mean I just shouldn’t drink it at all?