help ladies!

I need some advice here; in two minds with what to do, 
We've recently just moved house, Me my partner and my 6month old. The house is perfect We are in love with it. A lot bigger than our previous home and can totally see my son growing up here running around etc
We have an amazing landlord who is willing to meet all of our requests and has even gone the extra mile by saying he'll fit us a security camera up and get a taller fence for us. 
Now we get to the bad parts & the reasoning for the camera etc; 
Our neighbours are not the best. They ample cannabis openly might I add with children in their house, they have had the odd argument (quote loud and vulgar language) don't get me wrong I'm not a prude but I have the bairn to think about. They hang out in their back yard whilst their friends come over drinking/smoking etc (not all the time might I add just every now and then) 
I just don't know what to do for the best cause like I say it's an amazing family home just the neighbours on one side are the issue!  
We have told the landlord we are unsure if we are staying due to the reasons I said above, and he says he will do anything to maksure we feel safe and secure in our home. 
What are your thoughts??!