Husband Hates When I Go Out

So first & foremost, I'm a homebody. I rarely want to leave the comfort of my couch..RARELY. I'm from "small town USA" so in my teenage years, what we did for fun was ride around with friends on gravel roads. Embarrassingly enough, when I do get the urge to go out, this is what I enjoy to do. I have one girlfriend that I am constantly in contact with which is who I go "gravel travel" with. My husband & I have never had any issues & usually do a lot of things together, he doesn't like to go "gravel travel" so on the rare times I want to go, he throws a fit. He tells me it's stupid & that he can't go to sleep without me. I don't ever go out & do anything & if I want to go "gravel travel" I want to do it with my girlfriend & not have him tag along. I want us girls time. Am I in the wrong for this? What do I do about him freaking out?