Need advice!

My husband wants me to quit my job. My mom watches my son a majority of the time. When we go out, work, etc. She has a few health problems and also a life of her own. Daycare is not for us so we have both decided I should quit and stay home (I work a few days a week for beans). I am scared to quit. I don't want to have to tell my boss... I know.. silly... I'm afraid I'll hate being home. I'm afraid of going crazy all day without other adults to talk to. I'm also afraid of having to be dependant on my husband for money, etc.