Question for mommies out there.

*really long, sorry** I have a beautiful smart three year old little boy. His father and I are no longer together and I have sole legal and physical custody. I used to let my ex take our son a few hours a week but due to safty concerns they are no longer allowed to be alone together. I feel family is important and have always tried hard to keep his dad and grandparents in his life. The thing is is that the last year his dad couldn't care less if he gets to see him, and my ex's parents are demanding and disrespectful when talking about me. They always ask to keep our son for a week or more at a time and FREAK OUT when I say no. They say I'm trying to cut them out of his life and that Im inconsiderate. I don't HAVE to share my son, I do because I'm nice. They still see him 1-2weekends a month or so, but every time they give me hell saying that he will "foget who they are" and its not fair my parents see him more ( its none of their business) they live nearly an hour away from me and act like they life down the block. I know they love him but its reallly starting to strain me. Its hard and its wearing me out. Do you think I should give them less time with him (ideally its what I'd like to do) or do you think I'm not being fair and should let them see him more? How do you set up and handle sharing?