Help!! Legal advice!!!

My son is 5 months old. I'm single.

My sons father takes him once a week for overnight stays.

I was fine with it for abit. As we used to both Co sleep with our son.

He is now 5 months and rolling!

He has his own cot he sleeps in as it got dangerous (even putting pillows, as he pushes them off to bed)

My ex doesn't have his own room, also no bed, he sleeps in the lounge room with a mattress layed out on the floor. And shares the house with 6 other younger siblings with all have rooms to themselfs. My sons father is 20 and lives with his mother.

My son has no routine there. As the kids are all awake on the weekends, there's no school the next day. And watch TV in the lounge room late (where my son is meant to sleep) he also has no cot in the lounge room and just sleeps next to his father on the lounge room floor.

When my son is in roution he has 2 small naps a day and sleeps a good 5 hours at 9:30 ish,Sometimes even 5.5 hours.

Ever since overnight stays he is difficult to manage! And I have to work with him again to get him into routine.

It's difficult for me as I'm in never been comfortable with overnight stays, especially when he takes my son to his mother's house which is 1.5 hours away.

There has also been issues where my sons father does not know how to settle him and will ring me constantly till I pick up and help him sort out the issue.

What can I do about this? There's so many reasons why I'm uncomfortable with him staying at his father's now.

Safety concerns, and his out of routine.

I would like to stop overnight stays but do I have any legal ground to stand on?

(Also would like to add I want my son to have his father in his life, I have told him many times he is more then welcome any day of the week to pop in whenever)