TTC and changed my mind

I was really excited for months to start TTC and now that it is taking a while, and I've had some major life stresses come up, I have been lately feeling like I can't handle having #2.  I don't want to disappoint my hubby and daughter, or my future self by not expanding our family just because I hit a rough patch- but I can't help but starting to feel this way. Anyone else?