Would it be wrong to tell my husband his dad can’t come over unless he’s home?

My husband’s parents live an hour away. They work full time but come over almost every weekend. His dad likes to help my husband with yard projects and stuff. Since my husband and I bought our house together, it’s become a bit of an issue because his dad thinks and acts like he owns the house with us. I love to host them and make them feel welcome, we even have a designated guest room for them because socially we enjoy having them here. But last weekend they spent the weekend here. Around 1pm Sunday I go outside to find my husband alone outside and his parents gone. He didn’t know if they were coming back, they basically left in a huff because my husband didn’t agree with his dad’s advice on where to put our new shed. It became a giant argument that lasted into the night via calls and texts. My husband kept trying to move on but his dad would send these paragraph long texts and voice memos rehashing it and talking to him like a child.

This week my FIL has off from work and assumed he’d come over tomorrow to work in our garden and I don’t know what else. This was never discussed with my husband and I had already planned with him that he can come but I want him to be home too so it’s not just me and his dad all day (I’m on maternity leave with my son at home) and so that my husband could be with him in person instead of my FIL constantly calling him at work asking questions and arguing like I know will happen. So my husband told him that later in the week is better but FIL flipped out and it escalated to a big argument again where he was insisting that my husband agreed to let him come over tomorrow and husband kept trying to deescalate the argument and move on but FIL wouldn’t allow it.

Now my husband is asking me if his dad can come tomorrow but he will be at work bc he can’t take off at such late notice. I’m fine with FIL coming over tomorrow if my husband would be home but he won’t be. So is it wrong for me to say no? I really hate when his parents treat him like a little kid and throw a fit to get their way. I don’t want to be home alone with my FIL because I’m really uncomfortable with how they treat us lately. My MIL told my husband that we would be stupid to not let him come over tomorrow bc he was already planning on it and is upset now. So now I take it as she’s calling us stupid which is even more awkward.