I used to drink milk a lot now I can't...

So I recently stopped drink cow milk. For a while now I've felt this uncomfortable fullness and ache in my stomach after drinking it. As a child I drank milk quite often. Mostly at night and morning. I switched to unsweetened vanilla almond milk and now I don't have that ache in my stomach after having a glass of milk. Sometimes I taste a bit to much almond but it's a good replacement. I'm not sure if I'm lactose intolerant. But my brother is. He can't have to much or his stomach aches. He can have a glass but not as much as a normal person. Has anyone else experienced this problem despite being a big milk drinker for years?