cat lovers I need advice

so at my first ultrasound / dr appointment the doctor asked me if I had cats and I said no and he said " good " I asked him if it's bad and he said yes but I forgot why. I asked my uncle since he's a nurse and he said it is bad to have a cat around when you're pregnant . well , yesterday my boyfriend bought a baby kitty it's 5 weeks old and i've been holding it , petting it , and feeding it . 
the thing is i'm allergic to cats but I've never told anyone that I am so he was like " why didn't you tell me ? " . Everytime I pet it or it's around me my nose gets stuffy and I have to be blowing it alot also I sneeze a lot and my eyes get a bit red . I'm worried that this kitten might hurt my unborn baby in any way , i'm currently 16 weeks pregnant . the thing is , my cousin always had her cat around her during her whole pregnancy and her baby is just fine . 
I looked it up online and it said something about toxoplasims in there poop ( correct me if i'm wrong ) 
I just want personal experiences , is it bad to have a cat around you or is it not something to really worry about ?