Stress

I have been having a very stressful pregnancy due to relationship and financial issues among other things which are out of my control. I try to stay calm and do my best but I am worried about the baby constantly.... I am 20 weeks. I read alot about how anger and stress can negatively affect the baby but when I asked my doctor she said not to worry that it won't hurt the baby. Obviously she told me to try and remain calm... The less stress the better pregnant or not. Please share your thoughts.