Fiances family is pissing me off

Me an my fiance an our 1 year old son live with his grandma an grandpa an mom an My fiance's family is always saying oh this is whats wrong with him oh this oh that an always wanting to have him upstairs for hours an if i say no they throw a bitch fit an gang up on me. Saying im keeping him from them an always thinking they know my son better its starting to irritate me im his mom not them what do.i do ??