TTC overweight

Does anyone else get kinda upset when people say "maybe you're having trouble cuz you're overweight" 
Obviously I am overweight. It is no shock or surprise to me. I can honestly tell you if I had a dollar for every time I've been called fat, I would be a MILLIONAIRE. 
However, I am TTC. I understand weight may be a slight issue, I AM working on it..BUT I also know girls TWICE or even THREE TIMES my size who have multiple children or are pregnant now!