Sleeping habits

Baby is almost 4 months old and he can put himself to sleep which is good but he can only do that if he has his pacifier in his mouth! Sometimes he will wake up in the middle of the night and if he spit it out during the night he will look for it and won’t fall asleep if it’s back in his mouth which causing me to be up and down out of bed a few times just putting it back in his mouth! Idk if i should start having him sleep without it or wait longer but I’m going to go nuts with this damn thing ! Especially when he purposely pulls it out his mouth 🥱