SO leaving me alone after having baby

So our daughter is only 5 days old we only staid at the hospital one night an he was constantly trying to leave he wasn't there most of the time he didn't miss the birth but after I had her he kept leaving our second night home her 3 days old he left with his brother till 4 in the morning an again today he's leaving to go fishing with his brother we also have a 3 year old he's home most of the time but we just had a baby an I don't want to be left alone with two kids yet I need help an I told him this the other day when he left but obviously he doesn't care