Possible milk allergy anyone else experience this?

My son just turned one a little less than a month ago. He was on Similac sensitive formula. Which is for "lactose sensitivity" so it has way less lactose. He was on that because regular formula hurt his tummy/gave him gas as a small baby. When we switched to milk he had diarrhea. We tried it for two weeks and it continued. I told his doctor but he said drinking whole milk shouldn't be that different for him because his formula had the same things in it...? But to try something else like soy or almond.

I opted for the lactose free milk because I want him to have the right nutrients and that is the closet to whole milk. Welp, diarrhea since like Thursday/Wednesday (first week trying the lactose free milk, he was good for like a day without diarrhea) So now probably gonna try almond milk.

But what could be doing this? I give him cheese ALL the time. And he's fine with it. It's just the milk.