Coping after adoption

Not sure where this should be posted, been with my partner nearly 3 years, when we meet I knew he had a child, and learned he didn't have contact he was actually trying to stop his son mother from adopting him out.. They both have very rough childhoods and teenage lives, I completely understand why she would feel it's better to give their son a better chance in life.. Now a few years on my partners family still boast about this child as though it's their grandchild, let say the babies name is John, john has been with his adoptive family since birth has only known them since birth, they all talk as if John is there family. I can see it hurts my partner when people ask about his son and when he sees photo ect..

Little extra history me and my partner did try to fight for John as I was in training to become a support worker.. John was lucky enough to land an adoptive family who were lawyers, me an SO decided this was the best environment John would ever get as we couldn't not offer such support yet... Fast forward a few years on and we have decided ttc I feel like it's time we as a family including my in laws should speak about John and how it's time to let him be free with his family, taking photos from social media ect.. I don't like to see him hurt when people stop him in the street asking about John.., we both think it's time to let go..,

Anyone in a similar situation offer any advice please?????