Husband doesn't seem too enthusiastic about TTC

My husband and I have been actively TTC for almost a year, but have been together for 6 years and still no lil bundle of joy. There are times I feel as though it's my fault we can't get pregnant ( we haven't been to the doctor yet to see if I can/cannot). As I explain my frustration, my husband just shrugs it off. And always says, well what you want me to say. I'm feeling really down about not being pregnant especially when there are so many women around me who are pregnant.