When should you start your nursery?

So we already have crib and dresser but will be painting it, however, we’re putting baby in the room closest to us so we’re moving our boys to a different room. When would be a good time to do this? I feel the sooner the better but after 3 of my cousins experienced losses at around 20 weeks in the last two years I’m kind of nervous. I’m 15 weeks right now