Too young to be married?

Hey guys, so I'm almost 21. My fiancé is 23. Both of us are in college and work full time. We both provide for ourselves. But I told him I didn't want o start planning a wedding yet until we are a little more settled. I work as a waitress so I see a lot of people come in and out, and a lot of them comment on my ring. Comments like "yeah that's a mistake, sorry sweety your too young for that to work out." Or things like, " yeah I've been through three divorces, you'll see." It honestly makes it hard for me to be polite to these people but it is apart of my job. Is there really that much of a problem getting married younger? My mom got married at 22, my grandmother got married at 16 and neither have been divorced?