Is this my fault?

My first time having sex and his erection kept going down. I’m wondering if I was doing something wrong. By the way this was not his first time, and he seemed irritated that we had to keep going back to 4-play to get it back up (I assume that hadn’t happened to him before)