Turned off by his low sperm count

I know it's not his fault, and I would never let on or tell him but, I feel a little less attracted to my husband since finding out he has extremely low sperm count. There is something innate inside of me that feels turned off knowing he can't get me pregnant. I know if the tables were turned he wouldn't feel any less attracted to me, which makes me feel even worse. I feel horrible guilt for feeling this way and wonder if I'm the only one?