Pregnant with # 3 and husband would like to try for #4!

We are pregnant (due date April 3rd) with our third child and I know that my husband would like to try for a fourth, even though he hasn't said it directly, but my husband really want a boy so his family name and legacy can keep on going. We were hoping third time is a charm, but our third is another girl. 
I read other women's post, who are having/have their 4th or 5th children. Sometimes I think, I can support another child and I would like to give my husband another child, but 1) I am a planner; 2) with me being 40; and 3) with today's economy I don't if we can afford to have a fourth kid. Financially, we are both doing great; however, you never know what and where life might take us and if we add a fourth child it will not allow us to give to the other children that we already have and about to have, towards their college funds, and savings account and to include mine and my husband's own retirement funds. My girlfriend, once told me we will never be financially ready for a kid, but it shouldn't stop us from having our family. 
What do you ladies think? Should I try for a fourth and hope that it is a boy?