Sex after birth

I’m 3 weeks PP and really want to have sex with my fiancé. But they say wait 6 weeks after giving birth to actually have sex. Do I have to wait the full 6 weeks? I didn’t tear during birth and had a pretty smooth labor. Would waiting until 4 weeks be okay!?