Older men....

I'm 31 and recently started seeing my friend who is 50. Well, a friends with benefits sort of relationship. It was all fun and games at first, but now we are falling for eachother. Almost to the point where we want to say we love each other. Here's the problem.... He isn't exactly someone I can take home and introduce to the family. For one, my mother wouldn't approve and my older brothers would probably think it's disgusting. I love my big family but I don't think very many of them would approve of this. I know age is just a number, and I know you can't help who you like. But we have been out in public a few times and gotten some really weird stares. We are both not bad looking people and we are both fun and have good personalities. I feel like we deserve eachother. I need some serious advice.