should i care about my relationship with his family

My fiance so close with my family i invite him to literally everything he's always around they talk like nothing he's basically family, but when i went over for thanksgiving i feel so awkward its so quiet like im the gf he just bought home i felt 
more awkward when some of his siblings walked past me and say hi to his brothers babymom and ive been around longer and I shouldn't feel that way weve been knowing eachother 4 years talking for 2 and engaged for 1. And i just dont get why he only brings me around during holidays but yet we're always with my family i wanna build a relationship like hes done with mine and feel comfortable when i come around and idk if i just shouldn't care what they think or how close we are or think of ways. My anxiety bothers me in these situations