He's miserable and it's killing me

I'm currently 9 weeks pregnant and my s/o and I tried for 2 years to conceive this baby. But shortly after finding out I was pregnant he told me he needed to leave and he did... until later that evening when he came back home. He won't exactly admit it but I know he only came back because I'm pregnant. But now everyday with him he's miserable. I can tell he's depressed and doesn't want to be here. And I guess i must be selfish because I'm too afraid to do this alone than tell him to leave. But this was supposed to be a happy time. We were supposed to be experiencing this journey together but now I just feel like a burden and a little ashamed that I'm pregnant. He's happy some days or for half a day. Then a switch flips and he's closed off, looking pissed and won't really talk to me. I just wanted this to be a happy time.