Should we have sex?

Me and my boyfriend have both been with other people before and we want to have sex but we think it’s too soon. We’re both clearly ready but we’ve only been dating for 3 weeks. We’re worried that the relationship won’t last if we take things too fast. Any advice?