Scared

I have recently gone off most dairy products. I have been told before that soy can effect fertility, but then heard it doesn't, I've googled it, and have read it does effect fertility, I was having it in my tea and in my meal replacement shakes, which I was having 3 times a day for about 2 or 3 weeks. Has anybody else heard this? Should I be worried?