MIL with baby?

This is probably a stupid question but my son is 4 weeks old. Ever since we got home from the hospital, my mother in law (who we live with) constantly wants to be the one to take care of him. She's always looking for excuses to get me away from him and literally spends 16+ hours with him because I'm too afraid to put my foot down and say something in case I upset her because she's elderly. It's even affected my ability to breastfeed him and honestly, it's starting to seriously depress me. Anyway, at this young an age will he create more of a bond with her than with me? Is she going to replace me in his eyes? I nurse him when I have him but that's becoming less and less... I hate it and I'm even afraid to tell my husband because every time I mention it he blames it on my hormones.