Dad not bonding with baby

Our LO is almost three weeks old. DH works 5 days out of the week and is gone about 14 hours out of those days and I believe it’s putting a damper on the bond they should be building. My maternity leave is unpaid, so we can’t afford for DH to take extra days off. He does physically demanding work, so he’s tired when he comes home and since I’ve Been home with the baby all day, Ive told him to just rest and I’ll do most of the care for baby. If I’m cooking or just taking care of my own needs, DH will fix him a bottle and feed him, change him, and cuddle him. However, it’s not often and it’s probably my own fault. Maybe I’m not allowing (that sounds terrible, I know) DH to do more for baby. We are both first time parents, but I’ve had more experience with babies, so whenever LO starts crying while DH has him I’m instantly trying to take him away. I try to wait to see if they can work it out, but LO is a really angry crier and will do breath holding spells that send me into frantic mode and I take him from DH. Once I take him, he stops crying almost right away and I can tell it makes DH feel terrible that he can’t comfort baby. I know this early time in LO’s life is important for bonding and I feel like I’m ruining it for DH, but I just can’t figure out how to loosen my controlling grip.