Baby's middle name controversy

My pa's last name was "prince". We were incredibly close and I looked up to him all throughout my childhood until he sadly passed away around 8 years ago. This is my first baby and I'm having a boy and wanted to incorporate his name into my baby's name. I love unusual or uncommon names and his last name has really stuck to me, however everyone seems to have something to say about it because it's "prince". It has nothing to do with royalty. I've kept the first name from everyone because everyone carries on when I chose something I like. What I want to know is, do you think the middle name "prince" is silly and should I change it? His name would be Harley Prince.