SO's Parents Expect to see baby more often then I'm comfortable with.

So my SO parents, the grandparents of our child have this idea in their head that they have to a see the baby all the time, like every week. They can only come up on weekends because my father in law works during the week which is very inconvient to expect that every week when weekends are usually when people do things and usually when we're busy. When they come they visit for less then an hour so it feels like such a waste of time to clear our schedule so they can barely spend time with us, I would prefer if they visited less often for longer periods verses frequently and really short periods. We live about an hour apart. And the past couple weeks we've got pretty busy on the weekends with going down the shore and having friends over. They are mad at us that we haven't seen them in a few weeks and it's causing some tension. I also have had some issues with going over their house and with them in past because our beliefs are very different and I also get really bad anxiety when I go over to their house. Also they have a lot of pets like birds and dogs and other pets as well and I don't really feel comfortable bringing my baby who is less then two months around that many animals it doesn't feel clean to me and their house in general isn't very clean. I understand they are the grandparents but I feel like there needs to be a boundary that I'm comfortable with and that is more convient for us with a newborn. They act like I don't let them see the baby, but our schedules just don't clash well and they don't visit for long at all when they do. I feel irritated that they have this expectation of coming over all the time when they know to begin with that I have anxiety about being over their house and around them in general and that we get busy on the weekends or atleast we have for the past two weekends. Am I unreasonable for thinking they need to take what they can get with seeing the baby and back off a little?

Edit: we don't have plans with them every weekend, weekends are just the only days they will come over. So sometimes on weekends that when our friends are free or we are busy doing something else