Christmas away from home?

We just moved to a different state, 6 hours away from where we used to live to be closer to my family. All the years before we would go to my boyfriend's grandparents house for Christmas eve and spend Christmas at home, which I was perfectly fine with but now he's informed me that were still going down there for Christmas eve and Christmas.. uhh what?! That's along way from home and to be honest I would prefer to spend Christmas at home. Not only that but the drive to his grandparents house in December can get pretty nasty with snow and ice and we have a 6 month old..when I told him I didn't want to go he told me I was "bullshit". I told him if the weather's fine we can try to go a week before Christmas or something but that's not good enough for him.. am I wrong for not wanting to go? I just feel like Christmas should be spent at home?

When we lived away from my family we did go up there every Thanksgiving but I feel like that's a little different then Christmas.. am I wrong in thinking this? I don't mind going down there for a holiday but to me Christmas is meant to be a special time of year and I would prefer we spent it at home with our son. Honest opinions please!