Late preemie still sleeps all day 5 weeks adjusted age

Is this normal or should I encourage more awake time with him? He was born at 35w4d, 5lbs 11oz, was crying his lungs out as soon as he came out which means strong lungs, taken off the heat lamp in less than a day, bottle fed at the NICU with no problem, no jaundice. He is 9 weeks actual and 5 weeks and 4 days adjusted. When we took him home, that's all he did, eat then sleep. When he was nearing his original due date, he sometimes no longer sleeps right after every feeding but cries then sleeps. Now he cries more in between feedings if a don't sing or rock him to sleep. If I try interactions with him, he cries and cries. What should I do. He is cow's milk intolerant according to his pediatrician and is taking a special formula and I've been off dairy and soy. I can't do tummy time with him and the grabbing exercise that his pediatrician wants me to do.