Sleepy but curious

My baby is 5 weeks and he wants to see. He will fight sleep and if I let him, can stay up for 4 hours. Just looking at stuff and listening. He doesn't want to miss a thing. At the end he is just fussy and crying. I never let him go more than 2 hours without a nap.

I breastfeed him, change diaper, lay down with him and darken the room so that nothing steals his attention. Then he sleeps. But if I dare get up. He will wake up again.

How do I get him to continue sleeping without me? He sleeps through the night with me but won't last 5 minutes in his crib. Yes we started off well with. Crib but he started crying in it and I gave in.

Ps. Please don't remind me of SIDS, I am well aware and I have turned my bed into a big crib so...