What should I do?

Me and my SO love the Rory and have been trying to find middle names starting with the letter E. He suggested Elizabeth which is his mothers middle name. Me and his mother do not have a close relationship and honestly I do not like her as a person. I do not want this woman thinking that I named my baby after her. He is starting to really like the way Rory Elizabeth sounds together, but I just can't he in board. I don't know how to tell no and why I don't like this name without sounding like a total bitch. What do I do? And any other E suggestions?