Husband is Infertile...Anyone else?

Hello all TTC. While on our five year journey of infertility, I have noticed that most of not all people who find out we are struggling to get pregnant assume the problem is with me, the female, and start immediately flooding my ears with annoying and irrelevant advice for me to "fix" my parts. When I mention that actually my parts are working fine from what the doctors can surmise from multiple (and painful) tests, and that it's actually my hubby that is having the issues, they get very quiet and stutter a lot or just change the subject all together. What, no lengthy advice for us anymore now that you know it's the male that is technically the infertile one? Does anyone else have to deal with this? I get so tired of people automatically assuming my body is the defective one. Not that I want to push my hubby out there in the spotlight and point fingers at him (it breaks my heart to even think I might have done that in the past out of frustration), but people can be so ignorant and insensitive!