sex after baby..too soon?

I had our daughter on September 2. Our first wedding anniversary is October 3. I know they say to wait 6 weeks to have sex after baby. But I'm wondering what everyone's opinion is on doing it sooner. My husband is having a hard time with the no sex rule and it's only been 2 weeks! Id love to be able to have sex with my husband on our anniversary, but I also don't want to rush anything and end up hurting myself.