Am I in the wrong?

I bought my 21 month old a table for Christmas. The one that is for children with Frozen on it. Well, it’s been at my moms until just about a week ago. I also have a 3 year old. So, since I brought it back home, the kids cannot get along when they sit at it together. She pushes it at him, he hits her, they throw toys at each other from across it. It’s been like this since I brought it home. I’ve tried talking to them, telling them to stop, time out, etc. Yesterday, they did it once more & I took the table & put it in my daughters room. So when I get home from work, after telling their dad we were done with the table, he has it out with them eating at it. Of course, everyone knows children are better behaved for the dad than they are their mom. So they’re sitting there like little angels. Anyway, his dad starts asking why I put it in the room? I said bc they don’t know how to act with it so I put it up for now until they can learn. His thing is how will they learn if I take it. My thing is, they can go back to eating how they were eating bc it’s called punishment. When you continuously do something & after being told not to numerous times, it’s time for it to be taken. Just like a toy. According to him, I’m lazy & didn’t want to take the time to teach them. He says punishment is for prisoners & people who go to jail. They’re too young for punishment..... yet he puts them in time out & takes their toys. Not sure what he considers that. So am I wrong for taking the table?!