drinking while trying to conceive

I dont drink alot maybe have a 4-5 beers on the weekends...will that lower my fertility rate majorly?