husband and I are having major different issues

My side- I am the first to say that emotionally, I'm struggling post baby. I'm going to bring it up with my doc this week but I really don't have the time for therapy. Anyway, I'm back at work, have been since early November, and every morning I cry on the drive in and think "I can't do this. I can't be away from baby." I make it through everyday. I've explained to my husband that being away from her is actually painful for me. I'm going to take the summer off (I work in a school but they pay BANK if you work in the summer) as I really need more recover time. He's giving me absolute hell for it. 
His side-he does everything he can to get a break from our daughter. Always puts her down in the swing or exersaucer on the rare occasions that he helps out. He says we should take a vacation without baby. Says we should drop her at grandmas for an entire day or weekend to rest, hang out, go out etc. I'm ok with date night but not ready for all that. So I feel like I'm struggling and can't stay away from baby, and he just wants to be away from her. 
I know he's right and I need to eventually be less clingy, but I'm not ready. Anyone go through this?