Confused?

So basically I moved with my boyfriend early 2014 and we have been together for Nearly 3 years now. (we live with his parents btw)

He has a younger sister (15) and Brother (16). Ive always been really close with his sister helping her with school problems and etc but last year when moved to another house ive gotten closer with his brother and we all play video games, watch movies etc together ( as in all of us). I didnt see much problem with as i was trying to make a bond with his family have that trust etc.

Today My partner says how he's sick of his siblings having a better relationship with me but he mentioned how he's annoyed of how close his brother and i am. Im 18 and i honestly dont see him like that in any way possible. He's a child. I see him as the younger brother i never had.

I said id back off if that would make him happy.

But im just wondering if anyone has had this problem? I love my partner so much and i know he knows that. Im just shocked really. Obviesly ill back away though because thats what you do when your with someone. I just want to know if this is weird?

Thanks