Baby's Initals

So, we finally came up with a name we both love and I was really happy about having Oliver Jay. We've told our friends and family and a few have brought up his initials and a certain infamous ex-football player, which hadn't even crossed my mind. The middle name has been decided for years (my husband's middle name), and Oliver is one we both just really liked. I don't think it will be an issue with anyone his age, because they won't know the reference, or even after he's born (I couldn't tell you the middle name of most kids I know). So I'm not too concerned in the long run, but I'm not excited about adults who think they're funny/original making comments for the next four months. Is it too awkward at this point when people ask what his name is just to tell them the first name and avoid answering if they ask the middle? I'm trying to figure out a smooth way to do this as we go to a decent sized church and I inevitably end up having the same conversation 10 times in a 30
minute span. :)
Edit: Thanks y'all! Appreciate the comments. :) First for the reassurance, and second for making me feel like I'm not naive for not noticing that. Our last initial is B, so like I said, I'm not too worried for the future. It's just having multiple people make comments that's getting to me now. First pregnancy, so it might also be just getting used to all the comments in general. Thanks for your responses though!