Husband wants to take my baby to Virginia

So tomorrow is my brother's wedding, we've been planning on going to it for months. Well just last night my husband told me out of no where that tomorrow he is going to Virginia (we live in KY) for a friend's baby's birthday party and wants to take our 7 week old son.

I was pissed off because he knows I'm not going to miss my brother's wedding and he gave me the option of going with him or staying and he takes the baby alone. I'm not comfortable with that yet. My baby hasn't been away from me for more than an hour since he was born, let alone out of the state without me.

Well we ended up in a huge fight because he thinks I don't trust him with the baby. But really I just think it's stupid because he's not even close with this friend in Virginia. This is my BROTHER'S wedding, a one time thing. I want to be there and I want my son there too. And as I said I'm not comfortable with him going on a long road trip just yet. We exchanged some harsh words and he slept in the guest room last night and hasn't spoken to me since. Am I being ridiculous? Should I just let him take our baby out of state last minute?