Opinions please.

My son is 11 months old. And I really don't like leaving him. I work 3 hours a day Monday - Friday and he gets left with one of his Grandmothers for that. 
my OH parents try and push me to let them have him all the time. 
Where as my mum and dad are quite happy to come here or invite all of us around for tea. Rather than insisting on alone time with the baby. 
But when I need them to have the baby for reasons such as a funeral I had to go to. It has to be on their terms only. 
Where as my mum and dad understand that I am doing what I see best for my baby and do as I wish. 
My OH however sticks up for his mum and dad and doesn't see a problem with what they are doing. It is only me?