How do I leave my partner?

I'm thinking about leaving my partner. Long story short I'm sick of his shit, lack of affection and attention and his half arsed attitude towards parenting our 6 week old son. I'm in the UK not sure if it's worth mentioning. Both names are on our child's birth certificate however we are not married. Would I legally be able to take my child with me and nothing could be done about it (he couldn't call the police and say I've essentially kidnapped my own son) 
Thanks for any advice.