Right or wrong?

Laura
My son is 4 and visits his auntie (partners sister) every week. While he is there he sees his cousins and uncle. 
Recently there has been tension at the aunties house (between the husband, son & daughter and their partners) and we've have said to the auntie and uncle (when this has happened before) that we'd only let my son visit of the arguments stopped etc. 
but now the uncle has started being more violent and throwing things around the house, myself and partner have said we will not have our son go round there ever again, as they clearly did not listen the first time when we said about him visiting to them. But I know that the auntie, uncle and cousins will pull out the 'your stopping us from seeing him' card! 
Are we in the right as the boy's parents to stop him going all together? 
I'm just sick of the Argo from that side of the family! 

Vote below to see results!