Problems with Dad

Does anyone’s kid just not seem to do well with their Dads? My husband didn’t really want me to return to work because of covid (I work in a personal care facility that hasn’t had any cases) and to avoid daycare, so I compromised by just working my scheduled weekends and I’ll pick up half shifts here and there during the week. It gets me out of the house and away from the kids for a bit. But my son just gives my husband so many problems and I feel like crying. Even when I’m home he still sometimes will give him issues. He’ll cry before the bottle, during the bottle, after the bottle. He has trouble calming him down. I’m at work now and he said he’s had a terrible day because he just kept crying and even putting him in the swing didn’t help. We’ve tried having my husband care for him/feed him once he’s home from work because I thought maybe he was so used to me being around. I’m at a loss at how to make it easier on both of them. It makes me feel guilty that I even came back to work. Do you guys have any suggestions?