am I too over protective?

Hey! Need your advise. As I being too over protective? My son is 5 months old, my partner keeps saying my son needs to go to his grandparents more. My problem is one side don't make the effort with him atall and he doesn't really know who they are and cries when with them, and the other set tried to make him eat chocolate last week! I don't trust either of them to have him for longer than a couple of hours! But I need a break but I can't let him go!