Partner is distant?

Me and my SO's relationship was great during my pregnancy, I was practically living with him and we were always showing one another plenty of love and affection, now since our baby boy has arrived, it was fine the first 2 or 3 days or so, but my partner is gradually becoming distant, our son is now 2 weeks old and his father has started coming round to see him less and less and is now missing days, I'm currently living with my mum as his place is too small at the moment, I thought maybe it was the change as I was basically staying with my parter before our son was born for over a year, I thought he was losing interest not only by not really bonding with our son or wanting to spend a lot of time with him, but also when I try and kiss or hug him he makes an excuse or pulls away, this is confusing and upsetting. He thinks us moving in as a family is a risk as he's my first baby and my partner works shifts and seems to think I'd struggle, I am seeing my partner less and less and I miss him loads, he's always out at the pub a lot just now with his friends. I tried speaking to him about it and he said maybe things are going how they're meant to and I said I wanted us to be happy as a family and he said it has to be strong both sides, that strong bridge made of sand will fall, and when I then asked if he was still happy with our relationship and had feelings for me he ignored my question and hasn't spoken about it since. Everything has changed since our baby arrived and I'm starting to think he doesn't want to settle down. Any advice? I'm stuck at what to do. He would rather I came to visit him rather than him coming to spend a lot of time with our son, I spoke about our LO staying over the odd night and he said his flat isn't tidy enough, surely that could be fixed? I just need advice on what to do from here, I had bad baby blues and this situation is just upsetting me more than it should and I feel I'm missing out on quality time with my baby son because of it.

Any similar situations? Id appreciate any advice ladies ❤️ xx