ever started having sex with your SO then decided to stop?

My SO and I don't have the best sex life. We both live at home and he isn't risky enough to have sex in any unconventional places. So far this year we've done it once I think. Even on the rare occasion when we're home alone he'll often still turn me down. I mean he obviously has the right to not want sex but when we only get like 3 chances a year it kinda makes me feel like shit to be turned down. He's also very quiet during sex which makes me feel anxious and takes me out of the moment. He just doesn't ~do~ much to turn me on or even acts like he wants to have sex. I've talked to him about this before but it didn't really go anywhere. He said hes allowed to not want sex (true but not the issue) and that he's just not a noisy/ talkative person in bed. I'm thinking of just completely taking it off the cards because never being alone is an issue, but I also don't want to leave it until we're living together and realise I'm totally unfulfilled. HELP! I'm only 19-  too young to have a dead sex life!