I need advise! 💔

I have been with my boyfriend for 1 year and 4 months. We live together and I am very happy and very much in love with him. 
Okay so he needed a job and my brothers found a job and asked him if he would like to join them, we are from Texas and the job is in Louisiana, so he said yes and he has been working for a month now. They all room together & I came to visit him and my older brother told me that he has been living with my boyfriend for 3 weeks now, my brother said "he isn't worth shit" that I can do better. It really hurts to hear that, he's my boyfriend I don't want to hear these things from my family. I asked why and he said that he's just stupid and that I can do way better. He said that there is more but that he won't go into detail. My brother said that he doesn't love me, but how does he know? He's never around us. I just hate this so much 😔 I know my boyfriend loves me and we have plans for our future. 
I love my boyfriend so much, and I don't know what to do.. Should I tell him? Or should I keep it to myself? If I tell him it will be very awkward but to me it's already awkward. I just feel very bummed out. I just don't want secrets between us. Please help.Â