Is 20 years old, too young to TRY to have a baby?

I'm 20, me and my fiance have been together for 4.5 years and have been engaged for 6 months. We would really like to have a baby and have thought seriously about it for about a year now. mentally we are ready and financially we have saved up and over budgeted. We are prepaired and ready, but my only doubt is that I dont know what other people will think (which shouldn't matter, I'm aware). What are your thoughts? When is TOO young, and does circumstances make a difference?