Guy trouble ! HELP

Okay so this guy is super sweet and amazing but I live in a small town where everyone talks a lot of crap and gets in peoples business . He isn’t the richest lives in his trailer with mom but he’s only 19! He works two jobs and is a hard working man and me I am privileged with nice everything . Should I not be with him since were in two financially different spots and people will talk shit or do I say screw it and be with him because he is a good dude