Nieces/Nephews

I've been with my partner for almost a year. My sister has recently had a child and we refer to him as both my nephew and my SO's. Is this weird? Should my SO refer to him as my nephew or our nephew? People have told me that he's just mine and not my SO's as well. We all consider ourselves a family. What are your thoughts?