Standing up for myself

Can I get advice on what to tell my bf to let him know what I say goes for our baby. I don’t want to seem unfair and like I’m not giving him a choice but I need to stand up to him I hate going to his parents house because my MIL is really overbearing and possessive over our baby sheesh that lady never lets me breathe in peace when I’m with my baby she’s always after me taking her from my arms whenever she has the chance. Then he wants to go over around 1-2 pm and act all stupid and not want to leave , it’ll be past 10 pm and he’s in his parents sleeping... like umm can we go. He will not want to leave if itsnot past 10 pm and that’s just way to late my baby’s eating/napping schedule gets all messed up since his mom wants to be holding her from day to night and be giving her from arms to arms. I know it’s my fault his parents are like this I never set up boundaries which I should have and now they want to act like his mom is my baby’s mom and override what I say. But I know it’s time to put my foot down I just don’t know what to say to him, I feel like I’m her mother and what I say goes and that’s it but I don’t want him to feel like he doesn’t have a say ???? Like he never does anything for the baby I do everything for her which puts me in the right to be the only one who has a say right? Or ...?