Taking away the pacifiers

When do you think is a good time to take away the pacifier? When is too long? Have you had several children with one that was harder to take away from the other? My son will be 3 in May and I'm having the hardest time getting the binkey away. We've got it down to only nap time and bedtime at our house but he won't go to sleep without it. The reason it's so hard is because when he goes to his dad's house, he just gives it to him because he doesn't want to deal with the screaming. I've talked to him about it several times, but he thinks It's no big deal so theres no use. It's very hard to keep him on a routine when he's over there so regularly (every other weekend and Wednesdays) and his dad won't respect it. I've been told by his daycare teacher that he'll grow out of it and eventually not want it. I just don't want a 5 year old with a binkey in his mouth, especially because of it does to their teeth. Any advice would be greatly appreciated! But dont leave negative comments please. I understand everyone has different parenting styles but I'm not here to argue about it. Thanks!