Pacifier okay?

My baby boy is four days old, brought him home from the hospital yesterday and breastfeeding has been going well. However, we've had some trouble consoling him when he's upset. After checking all possibilities (dirty diaper, fixing swaddle, ect.), the only thing that seems to work is giving him a pacifier - which I didn't want to do because I read that it could affect his breastfeeding, nipple confusion and all that. Is there truth to that, or should I not worry about it? He's still eating well today, despite having used the paci.