Master bedroom upstairs/kids downstairs?!

Hey Mommas, I have a dilemma! Our master bedroom is upstairs and our two kids (ages 5 & 8 months) bedrooms are downstairs. They both sleep with us but it is getting to the point where neither me or my husband are getting sleep at night. The 5 year old sleeps on a blow up mattress which is not bad but the baby....well we tried putting him in a pack n play and when I lay him down and then go back downstairs to finish other things he will stay asleep and not move for hours BUT as soon as I come upstairs he wakes up within 5 mins. It’s like he senses me lol and he will NOT go back to sleep unless I put him in the bed with us, which leads to us not sleeping. My problem tho is I’m not comfortable at all leaving him downstairs in his room/crib even tho he sleeps better in his crib (experience from nap time) so I’m not sure what to do or wondering if anyone has been in a similar situation and how it turned out/works for you? I can’t help but to think of a fire or burglar and I just don’t know! Also, wanting to move our 5 year old downstairs too so got to get that in the works. I just hate having my babies on a different level! Sorry this is so long!