Lasting longer

So my boyfriend and I sometimes use condoms and sometimes we don't. But when we don't wanna use condoms he gets close to Cumming with in a few minutes. So he puts a condom on....sorry tmi. But anyways has anyone had this problem? He seems to last longer with a condom on. We want to stop using condoms but we also want him to last longer than a few minutes. Anyone been in this situation and what should we do?