odd uncle

My SO is 22, I just turned 19 a week ago. He has a 30 year old brother who drinks and parties and still lives at home sleeping on his dad's couch. Well since announcing our pregnancy a few weeks ago he's offered me alcohol. I had no problems with him being around our child until one day at family night a few of us were on the back deck and their 3 year old niece tried sitting on his lap and I heard him tell her no. I was like okay reasonable. Until he added "I'm drinking" does that sound like a potentially inappropriate context? Not letting a little girl sit on your lap because you're drinking?