House of 8.

Myself and my son recently moved into my boyfriend's house. My boyfriend owns his home and his brother and his brothers three children live with us too. My boyfriend's daughter comes home every weekend but stays at her mom's during the week. So during the week we have a total of 7 people in one home, and on the weeked we have 8. So, when I moved in I let my 5 year old son take one big storage container of his toys into our new home. The rest of his toys are in storage for now. We have lived here for about 4 months and my son's toys have dwindled down to practically nothing! I am so pissed. The other 3 children.are older than ours. They are 11, 9 and 7. They are so harsh with toys. They have destroyed, lost or left outside most my child's toys. So today I've finally had enough. I took the storage container out of the play room and all the toys of my son's that I found scattered around and put them in my room. If my son wants to play with them I'm going to make him ask. Am I wrong for this?? It's so damn annoying. They're old enough to know better. My SO daughter and my son are both 5 years old and they take better care of their things than the older kids do. Ugh and to make it worse my son is sad about it. Suggestions, comments thoughts??