Flacid (soft) penis

Okay. So, I have been having this issue with me SO. In the middle of sex I'll try and lube it up. So he takes it out first, but two seconds after he is out he starts becoming flacid.  
It's not because he isn't aroused. He definitely is. The sex isn't boring by a long shot. 
Is there any other explanation or any other women who have this issue?