Mad at husband

So I started back to work part time 2 weeks ago and I HATE it!! The whole time I’m there all I want is to be home with my son (he is almost 11 weeks). My husband has mentioned in the past about me staying home so when he got home on Friday night, I brought up how I’ve been feeling about work and that I’d like to stay home with my son. And he said no!! He thinks I need time to talk to “real people” because my mom kept saying that it’s good for me to get away from my baby once in awhile and I need to work. I feel like they are ganging up on me and aren’t even listening to what I have to say!! We spent 3.5 years getting my son into this world (he is an <a href="https://glowing.com/glow-fertility-program">IVF</a> baby) and I just don’t want to miss a single moment with him. Why is it so wrong to want to stay home?! I just don’t even know what to do now. I love my husband but I’m so mad at him right now for siding with my mom that I can’t even look at him! I feel like I’m stuck between a rock and a hard place and I don’t know what to do now.