Am I being unreasonable?

So I’ve been dating this guy for almost 3 months now. He’s a nice guy and different from any other guy I’ve ever dated. He’s 30 years old and I am 28, he lives with his older brother, the brother’s wife and their two kids. I live with my brothers and my 20 month old twins. The issue I’m currently having is that I feel like I’m putting in all the work for this relationship to work, I am always driving to his house on weekends so we can spend some time together or I have to drive to his friends house to him there and hangout with his friends then later go back to his house. He never makes an effort to come see me I always have to go to him. I spoke to him about it and he said he’s worried about my brothers and dad’s reaction (my dad occasionally comes over to our house) if he comes to my house. But I feel like if he really wanted to see me he would just come over, we don’t even have to go inside the house, we can just hangout outside in his car (we only live 15 minutes apart) .

He never makes plans for us to hangout or go out somewhere. It’s just me going to his house every weekend and I am honestly getting tired of it, I’m starting to feel like he’s using me for sex and cuddling. I need some advise please, I don’t have anyone to talk to about this. I really like him but I’m having doubts about staying in this relationship. Am I being unreasonable?