Sleeping on tummy


My baby is 1 week old and he refuses to sleep unless he is in his swing (occasionally) or if he is on his tummy and he loves being on his tummy. I know he isnt supposed to but im always watching when he does just in case but how bad actually is it?