Fake Orgasim

One of my guy friends and I have been have sex. None of are other friends know since we just want to be fuck buddies. So, the first time it wasn't really expected and he wasnt able to cum. Which made me really self conscious and he knew that. After a few times I thought he was finally getting into it, but then the last time once he left I checked the condom (which I know probobly sounds weird) and it didnt have anything except leftover lube. Normally I feel like I can easily pleasure a guy.. but this makes me super uncomfortable knowing that he hasnt been able to finish. What should I do?