Won't hold him

My boyfrirnd doesn't like to hold our son. I told him to hold him earlier and he sighed as if annoyed so I said nevermind. He told me he doesn't need to hold him all the time because he knows he's his son but I said that LO doesn't know he's his father (he's only 3 weeks old). Like it's been days since he's held him, and he really only does if I'm in the shower or something and he starts crying. It just really upsets me.