Missing Boyfriend

Im having a bit of trouble with my partner, we spend almost everyday together, which I have never had someone want to be around in my company like that in any other relationship I have had before, so its nice, but the issue isnt that its when we dont spend time together, he always takes off to one of his mates houses to get drunk or stoned, instead of just chilling at home? I always ask is it cause he gets sick of spending time with me, when he isnt around me, he wants to take off and go wild? Am I holding him back from doing what he really wants to do because I dont approve of those things? Am I just being silly in thinking this behavior is even anything to worry about?