Grand parents rights??? UK

Me and my partner are splitting up due to his parents interfering all the time in mine and my son's life(it's a very long story). I'm trying to get a place of my own for me and my son. I've been threatened with court action by the grand parents if I stop them from seeing my son. But what are their actual rights??? Partner is named on birth certificate. Anyone have any advise please??