Husband keeps visiting his family

My husband keeps going back to his parent’s house at every vacation. Out of 4 weeks of vacation, I get to spend only 1 week with my family (he doesn’t accompany me) and ‘we’ spend 3 weeks at his place. I love to travel and would like to do that as much as possible till we have a baby. But my husband keeps insisting on seeing his family. It’s really troubling now because I don’t get along very well with them. And I have always been so much more attached to my family than he has ever been to his, but when you get married you need yo give all the importance to your new one and though he says I’m the most important I just don’t feel like it. I just can’t keep wasting my vacation like that. Talking has not helped so far. What do I do? How long do I keep following this routine?