My ex husband is being a bit harsh.

We have two daughters together. My fiance and my daughters have a wonderful relationship. I was blessed to have found a man that treats my girls as if they were his own. For the last year or more, my youngest has always called my fiance dad or daddy. We never thought anything of it because she is not even 4 yet. We dont encourage it. But even after correcting her time after time she still does it and we've given up correcting her, this is what she is comfortable. She is beyond stubborn. My fiances name isnt complicated but the baby just rather call him dad or daddy. My ex absolutely hates it because he's not their dad. Which I get that but hes threaten to punish her for calling my fiance dad. My ex often asks how I would feel if my daughters started calling another woman mommy and I honestly said it wouldn't both me. Because that means my daughters are comfortable around her. Our children are still really little. My fiance has been in my baby's life for majority of her life so far. Am I in the wrong for letting my baby call my fiance dad? Is my ex husband right that we should constantly be correcting her?