I need some advice PLS

Okay so I'm 20 years old and still living with my over protective dad. I currently have a boyfriend who is 26 (which my dad doesn't know about) and he wants me to tell my dad about us. Because he says that he wants us to move in together and get married and have kids. And he's always telling me that he will come talk to my dad if I'm scared to. (My dad has known him since birth) 
Anyways, I always tell him no. That I'm not ready to leave my dad, And my younger brother who's 15. I don't want to leave him, with no one to care for him. Also I think I keep saying No to him because I feel like if I tell my dad it will make this relationship too real. And I don't know if I want that, not because I don't love him. It's just that I feel like he isn't as invested or in love as I am with him. 
He is also into bigger girls and is always pointing them out to me saying "she's hot" and stuff like that. I don't really mind but I feel like if he starts looking he will find someone he likes better than me. And since I already feel as if he doesn't love me, it makes me feel undecided about our future together.
I've tried talking to him about how I feel but he is always telling me to think what I want, and stating  everything he's ever done for me. (He's gone against his family to date me.) and he's says that if that doesn't prove love than what does.
He also tells me that whenever I want I could leave him, and every time we fight he brings up something about us breaking up, and I ask him if that's what he wants but he says no. And says that he doesn't mind only having me as a sexual partner or sharing me as long  as he has me. Idk if I'm having these doubts because I don't love him anymore ? Or because I feel like I deserve better? I'm confused.
Anyways, i just wanted to vent and ask for opinions coming from another perspective. 
(A little backstory) I dated his brother who is now in jail on an off as kids, that's why his parents and his family and my family don't approve. I know that this was wrong, but it happened and we can't take it back.