Silly Age Difference?

My boyfriend is 19 and I'm 22. He'll be 20 early next year and I'll be 23 a few months after. I've already had a few comments about his age and me being older, do you guys think that age gaps are an issue?