Baby sleeping on his side - help!

Baby is a week old and keeps trying to sleep on his side in his swaddle. Anyone else have this problem? Should I be worried?