Is this fair?

My husband and I lived alone and then suddenly his mom and sisters moved in with us and they completely took over the house (I became very depressed) so we moved after a year, only thing is we moved 2 minutes down the road and now his mom gets upset that we don’t see her everyday and says that we don’t love her or care for her. I told my husband I want to move in between both of our families (30 mins away) so I can see my family like he sees his all the time. He got really annoyed and said I’m being ridiculous?