Breast feeding mama's...do you wear perfume?

So I know baby knows your scent and all that and I've heard before that you shouldn't wear perfume because it can throw baby off of your scent. I smelled a perfume I loved today and wanted to buy it but figured what's the point if I can't wear it. 
So, do you wear perfume?