baby naps

I just had my baby on Sunday so he is now 3 days old. While I was in the hospital I had very frequent night time colostrum feedings and after a while getting up all the time to put him back in the crib was a bit much with the pain so he slept on my chest. My husband, is starting to be overbearing to the point where it's irritating me. He constantly tries to tell me I'm going to kill our baby with him sleeping on my chest. Or he doesn't need to sleep with you he could get hurt. Me personally I'm fully aware he sleeps on my chest I most of the time don't fall asleep with him on my chest. We just got into an argument over this and he is acting like a know it all when it's his first baby too. I don't know what to do at this point.