when is it a right time to take away the paci?

Acouple of people told me one or else this happens? My son really just needs in for when he's in the car seat and when he's going to sleep. But besides that it's like it doesn't exist.