Fiancé and I hardly have sex!

We hard have sex and it’s pushing me away. I’ve tried for a year to be honest and open up and all he ever says is “sex drive is low” but never makes the attempt to contact a doctor. Yet he hurts a finger and calls a doctor right away. I just feel like if he was tired of hearing it or wanted to make it work he would. I do think more to intimacy than sex but that’s a part in it as well. I feel like I am fantasizing more and more and it’s making me sexually crazy. He says all day how much he loves me and cares for me but he just doesn’t want to make the move to make this better and that worries me. I’m also worried I’m the only girl this has happened with. (He is 26) we are having major problems because of this!