FTM distancing myself

I’m a FTM. I have a 2 week old baby girl. And currently I am staying home while my husband works. I go back to work in September. Anyways, when I take care of our baby girl during the night and morning (till dad gets home at 2pm) for some reason it always gets chaotic once dad comes home. And not so much baby girl but my husband and I. I guess we both get frustrated and at the end of the day we try to come together and support one another but as soon as he comes home I honestly am not in the best mood. Has this happened to anyone else? And I’m all honestly I feel like I am the one distancing myself from him… idk why. I do currently feel like I’m a mess so maybe because of how I am look and feel?