Milk and Me

This may sound nuts but milk may be a key in my weight gain. I LOVE milk! I'd drink only milk if I could justify it. Recently though, I think I've become lactose intolerant. I switched to Lactose free items and I'm down about 6 pounds. I may be grasping at straws here but going from full fat milk to much lower calorie forms like almond milk seems to have helped. Anyone else feel like changing their dairy intake helped them gain control over their weight again?