I don’t want to move?

My husband and I are 25 and 24 with a dog and no kids. He’s always said he wants to move out of state but I don’t want to. I want to be near my family and I love my job (they’re family to me) and I love my area. I couldn’t imagine up and leaving and I don’t want to. When I told him I’d rather not until maybe later on in life he jokingly said “well I’d just leave still”. Is that a thing? What if he’s dead ass wanting to move and I’m dead ass not wanting to?