Living at your parents with baby

My husband and I are off birth control knowing we very well may become pregnant. We have been married five years and both agree we would have had 4 kids by now if we could. We are both 26 and I feel like fear of having everything perfect like our own house has kept us from starting our family. My parents have let us stay with them for a very small amount per month (~400). In the meantime we have been able to start really knocking out student loans. I feel I would be ok with having a child at my parents which they would also be ok with while we save for a down payment. I am tired of waiting for all of my circumstances to be socially acceptable before I have my first. Would anyone else do this or has anyone else done this in order to start the family they've always wanted. Thanks so much for your answers!