Condom stretching during sex?

So my boyfriend and I have sex frequently but recently we've been having sex harder. We noticed the condom gets stretched out a lot after. It doesn't fall off or break, but I'm worried that it might end up tearing or breaking. Is this normal? I'm not on birth control but I hope to be soon. Should I take plan B just in case? My boyfriend is allergic to latex so we've been using lifestyles. Should we try a different brand? I'm probably thinking too much of it but I want to be as safe as I can.