Soya Milk good or bad for you?

I really need some advice! I understand that too much soya can be bad for you but surely there is so little in soya milk that it can't be too bad if you only have a small amount each day! I'm intolerant to all dairy and love frothy soya milk in my coffee each day, I make it myself and only use about 100ml! Now I'm TTC I'm worried it's bad and don't know whether I really need to stop having it! Would really appreciate any advice