Me, my baby and my work

Emily
I'm 14 weeks and 3 days with my first pregnancy. I have been blessed to both be pregnant and have had such an easy time so far. My husband and I have just purchased out first (and hopefully) forever home and have been busy decorating and renovating. 
I'm 29 (30 next month 😉) and extremely career driven. Despite this, having a family and a balanced work / family life is really important to me, and my husband. 
Now here is the clincher - I don't intend to take any maternatity leave other than a month to physically recover as much as I can (while keeping an open mind of course that I may not be ready etc. etc.) and bond with my baby. Instead, my husband is taking the maternity leave and we are both hoping to hire a reliable and trust worthy live in nanny to help us when the baby is older. I also plan to only go back to doing 4 days a week to allow me more time at home with my family. My work are fully supportive of my decisions and considerations however - no one else is. I have never felt so judged and scorned. 
People seem to find it normal and acceptable for a man to balance this but why not a woman? 
I still intend to try to breast feed and express for when I'm not there. I work 10 minutes away from my office so never far away to dash home. 
I do have a demanding job as a sales and operations director but l also have a great and hardworking supportive team. 
I understand that things will be hard and difficult to adjust but just because I love my job and am passionate about it, people seem to think I'm not fit to be a mother or would not love my child? 
I know I have some perhaps unrealistic expectations and things will undoubtedly challenge me to change and adapt but what's wrong with a woman going straight back to work? 
Do any of you with children work or expecting mums plan to return pretty quick? 
What are you views on woman going back early? 
*no trolls please*
Thanks in advance for constructive advice / comments. 
Xxx