Housing situation advice please?

I’m 25 and my boyfriend is 24. We currently do not live together and can’t for another year or two. We have a 3 mo son and he goes back and forth between houses. It’s usually every 4 days. I try and keep the routines at nighttime the same and he always sleeps on schedule, but I’m worried as he gets older he is going to get confused and will want a more “stable” environment. He has the same nightlight, crib, white noise, bathtime, etc at both houses. I try and keep it as consistent as possible. Is there anybody that has been in my situation or can offer some advice? Do babies even care? Please help me, lol! Thanks 😊