Little one spending more time with one set of grandparents and not enough with the other..

I know this happens a lot but I feel bad. Due to my husband getting out of the military, we live near his family so our little one sees his dads parents more than my parents. When we visit mine, my little one is scared and acts like they are strangers. Due to financial and work obligations, we aren't able to visit my parents for our little one to get to know them better, very often. I just feel sad that my little one won't have the same bonding with my parents that he does with my husbands parents.