My 4 month old is still being swaddled....

anyone else? He has only rolled once from tummy to back and that was with major help. He just adores being swaddled and has slept 9-11 hours per night since 6 weeks old. I know we will eventually have to break the habit with a slow transition to no swaddle, but I am terrified of that day. When should I start to wean him?