Why do we all stress about our diets when TTC?

I understand that a well balanced diet and moderate exercise is good for a healthy pregnancy and baby. I am by no means suggesting we don't eat healthy and don't do what's best for ourselves and our babies! My point is, there are women who drink coffee and energy drink all day, smoke packs of cigarettes a day, alcoholics and drug addicts who get pregnant. Whether they stop their addictions when they get pregnant is irreverent because they GOT pregnant with their dangerous and irresponsible habits. Why should we 'normal' woman stop drinking our 1-2 cups of coffee in the morning, not have a glass or two of wine a week with dinner, watch our chocolate and tea intake to be sure it doesn't have too much caffeine? Why do we stress so much over making sure we get the 'required' amounts of fruits, veggies and proteins in our daily diets and make sure we get in a light exercise almost daily? If drug addicts can get pregnant with filling their bodies with poison, how much does caffeine  and exercise honestly affect our inability to conceive?