Baby's bedtime after 3 months?

I've read after the 3 month mark a baby should be going to bed around 7-8pm. At this point our son still does to bed anywhere from 8-10, it really just varies depending on when his last nap ends and I go to bed when he does. Should we be more consistent with bedtime or just go with the flow? I also don't know how I feel about him going to bed around 7 when he's already awake by 5-6am every day and I would say sleeps poorly after 4am. If he has an earlier bedtime doesn't that make for an earlier wake time as well? I would looooove for him to be in bed by 7 and to have a few hours with my hubby, give him a dream feed at 10 and then have him sleep until 7am but that's just not my reality. I get so sad over my night routine sometimes. I never get a moment alone with hubby because when baby sleeps, I sleep. He doesnt nap more than 30 mins at a time during the day so I absolutely need to sleep at night. I thought things would get easier after the newborn stage but it's not!