Sex without birth control

Does it matter if your not on birth control and you have sex with a condom?? Could anything bad happen if the condom breaks?