fuck buddy

So my friend and I who had sex once before decided to be fuck buddies yesterday when we met up. He really wanted me to come over and He was moaning when I gave him a bj and when I was riding his face was priceless, total confidence booster and I was super into it. Both times we hung out he goes soft when we're trying to switch positions. Is this normal? I tried to give him a bj to get it back up but in his words "it was past the point of return". Is it something I did?