Socialising with work colleagues outside of work

So, my work colleagues enjoy going out drinking and clubbing which isn't my thing AT ALL. I don't like clubs and I don't really drink a lot. I'm not really that close with many of them so I know I'd feel awkward if I went, like I wouldnt have anyone to chat with, I feel like I'm kind of outside of the group. They all do the same job and I do something different, so all of their conversations are about their job and I can't really join in.

Sometimes I feel bad for not making the effort and going but then people say to me I shouldn't and I shouldnt have to justify why I'm not there. Christmas meals etc I always go to, it's just going out drinking.