My bf gets soft when I do stuff to him

I’ve been seeing this guy and whenever I do stuff to him he gets soft. If he is kissing me or fingering me he gets hard, but once I’m doing stuff to him he gets soft. He says I’m doing it perfectly it is just what happens with him. I’ve been with my fair share of guys and this has never happened to me so I can’t help but think it is me. I’ve asked if I can help in anyway and he says no. I have yet to have sex with him because I really like him and want to wait. Has this happened with anyone else? Any suggestions?