Stay at Home Dad stigma and judgment

In writing this, my knee-jerk impulse was to write a lengthy and detailed explanation to justify the decision my wife and I made to have me be a stay at home dad.  However, I feel like even that would be contradictory to why I am even writing this to begin with.  What I mean is, if I were a woman who decided to be a stay at home mom, I would feel no need to justify this decision. Yet as a man, I am faced with the stigma of stay at home dads which resonates from an archaic and patriarchal time period that no longer has any relevancy.  What is best for my family, financially and for my son's care, is for me to stay home.  My wife is at a point in her career where she makes gobs more money than I do, and if I were to go back to work it would actually cost us money in day care expenses.  Not to mention strangers would be raising my kid, and why would I want that?  So there it is, the should be unnecessary explanation as to why we, as a family, decided that me staying home is what is best for our son.  So why am I faced with this overwhelming sensation of being a failure, or that I'm a free loader, or that I'm taking advantage of my wife?  Are these things shared with stay at home mom's?  Do other stay at home dads feel like this as well?  These are questions which weigh down on me.  I know intuitively that this is what is best for my son, and I am excited about our soon-to-be adventures of hikes, beach walks, and shenanigans...but I can't seem to shake this looming feeling of being a lousy husband.  And why is that?  Why can't a man be a stay at home dad without these feelings of guilt or failure?  Why the need to explain myself?  Or am I just crazy and over thinking it?  Are there other stay at home dads who struggle with this as I do?  Do you stay at home mom's struggle with these issues as well?  I'm curious...