weed bf

i ask my bf not to smoke weed, he understands my reasoning and respects it. all of his friends talk major shit on me about how im controlling etc but its really not that serious. do they have a point or should i not let it bother me