Second Baby... or not?

My husband and I have a soon to be four-year-old son together. We had only ever planned on having one as he has two (much older) children from a previous relationship and we had a house full!
However... lately, my son has been asking for a baby brother or sister (seeing a lot of his friends and cousins with one), and I'm tempted, as is my husband.  We began trying for a second one, saying that either way we know we're blessed.
The thing is, when my son was born I suffered with post natal depression and anxiety, and although I moved past it, my current job (teaching) is also making me incredibly anxious and I have panic attacks a lot. I know that it's my job making me feel this way and the pressures of juggling my work responsibilities and motherhood, but my husband said this morning he doesn't want us to have another baby because he doesn't think I will cope.  
This has really hurt my feelings, but I know there is some truth to it. I struggle with one - how would I cope with two?  But at the same time, now that we started trying, I have my heart set on having a sibling for my little boy and another baby in my life.
Please give some sort of advice or guidance. I'm heartbroken.