Traveling with a newborn

I’m due May 11th. My husbands sister is getting married the end of June in Minnesota (we live in SC) The baby will only be around 6 weeks old IF he comes on or around his due date. My husband wants to drive with the baby and our 6 year old to go to this wedding. I’m worried about the baby’s immune system. I said we should fly by ourselves and leave the kids with my mom and go to this wedding. His whole family is pressuring me to take the kids. I just don’t feel like that’s responsible. My husband wants to fly and stay longer than I’d like to, I don’t wanna be away from our kids that long. It’s been a never ending argument. Give us some advice! Please.