so down

Named my son a name that doesn't fit. I thought he would grow into it and it still bothers me and idk what to do. My husband said we could change it weeks ago but we couldn't pick. Nothing seems right.  Feel like since we toyed with the idea and family thought we were just before, if we try again to find a solution and maybe change it, I will be ridiculed by people for doing this to him. It scares me. I just need to vent. He's almost 4 months old, it's too late to change it now......it's just causing some serious issues for me. I'm just sad I guess and wondering if anyone has been through this.
Edit: it's not an issue of the name being bad. The name isn't a bad name, it just doesn't fit my child.