Do you let this bother you or no?

My boyfriend is 27 and I'm 20... We love each other so much. I get nervous of our age and what people think... But I cant give him up... How do I deal?