Should we separate?

Ok. I need some advice. My husband and I have been married for 3 yrs. I'm pregnant now with baby #3. But something has been different. He has been distant since November when I found out about his affair. We have been sleeping in separate rooms. All because he said he wanna work on or marriage but he hasnt shown me anything like he wanna work on it. I mean how can you show me anything if you sleeping in another room and we hardly spend any time together since then. We literally go to work and come home and just be neutral. Like talk about the kids and stuff. But never about us. So I Recently asked him, do he love me and do he wanna keep this marriage and his answer was "idk. I'm frustrated and need space". So I'm like wtf. This not a marriage. We co-existing! I'm about to just pack my things and go because he aint showing that he care. He declines counseling. He will say he willing to go and blow it off the day it's time. And I arranged it 3x. Am I wrong for leaving?