Scared of miscarriage

I have never been pregnant before I am currently about 5 weeks..I have no family history of miscarriages or anything but everyone keeps scaring me saying I shouldn’t be excited just yet cause I might lose my baby..which is awful to think of and does not by any means make me feel any less stressed than I already am...I don’t drink caffeine I haven’t in months I’m not a smoker and I haven’t touched a drop of alcohol since I found out..Ive been making sure I’m eating healthy food every meal and taking my prenatals...I just can’t brush off what people have said to me...this is suppose to be the best time of my life but some how I feel like I’m not allowed to be happy