Parenting advice??

So my soon to be 4 year old goes over to his dads adoptive mothers house every other weekend. Sometimes they will keep him for 4 days. Any how, they haven’t really been the discipline type of people. I make my son mind. I have him pick his toys up when he drags them out. I do not let him sit and watch YouTube all day everyday, if you ask me, I give him too much screen time. If he back talks or throws fits he gets in trouble! I make him mind. Why would I not? Well for the past month or so he will come home from being at her house and he will not mind, he talks back. He tells me “ I just want to go to my Nonnas.. Or “my nonna doesn’t make me do that.” She picks my toys up for me.. a lot more. Yeah they are little things, but they really hurt my feelings and make me feel like he thinks I’m just being mean/ unfair to him. I feel like he would rather be over there than with me. Anytime I have ever addressed an issue with her, it’s usually an argument and then things never change. I’m just stuck on what to do at this point, I just need some advice on how another mom would handle this, I tend to jump right into things and I don’t think rationally. So I’m trying to be an adult about this situation rather than popping off at the mouth.