Going by baby's middle name?

My husband and I have agreed that we love the name Grayson, so that's what we plan to call our son! We agreed on the name Connor as a top contender for his middle name. However, he thinks that Connor Grayson flows better than Grayson Connor. He and a majority of his immediate family (parents, sister, a grandmother, aunt, and uncle) all go by their middle names. He's told me several times in the last few years that he doesn't want to continue the tradition as it can get confusing at times, but now he says he doesn't care about the occasional confusion. I agree that Connor Grayson flows better, but I really don't see the point of naming a child their first name and calling them by their middle, especially when it tends to set them up for confusion later on. Any thoughts?