Lack of intimacy

My SO and I have always had a lack of sexual intimacy. He says he gets very uncomfortable/nervous in the moments leading up to sex, and the anxiety causes him to always back away.

This leaves me feeling rejected, and sometimes just kind of stupid when normal fumbling becomes the worlds most awkward situation (since the whole thing is usually already awkward)

Im not sure what to do.. any advice?