Am I wrong or him?

So last year we had these neighbors, they were a bit older then me they were easily 40s, where as I am 25. My son is 7 and so is theirs. They both played together a lot outside of occasionally one of the boys went into each others house. The woman was real weird, she never would speak to me she had her husband do everything from talking to texting. The same way my partner always had me be the one. It’s just so awkward. So there was times when they wouldn’t allow their son to play with mine for months. Then all a sudden they would allow him and that cycle continued for a year. I never knew why? My son is very respectful and isn’t a mean kid. But as most kids there can be arguments and fights so that’s normal so nothing out of the normal. I could tell the parents never liked us. It’s just obvious. We ended up having to move, my son cried for their son. So I reached out through fb and found the mom and messaged her and asked if it would be okay if the boys found a day and could play. She said yes and kept it VERY short. She invited my son over to their house I dropped him off and the mom was in the house hiding. The dad finally opened the door and gave me his number and told me just to text him instead and he’ll message me when my son is ready to be picked up. I said thank you so much for having him he has missed you guys, he left me on read. I picked him up they kinda just rushed him out and shut the door. I never reached back out because i was the only one putting in the effort it all felt forced the whole thing.

I forgot to mention multiple different times when we lived next to each other they purposely ignored the door when my son went over to knock. A few occasions I would go over with him knocked rang the bell and I would hear the woman saying “ SHHH” to the kid to make it sound like they weren’t there so we’d leave. I can clearly see they didn’t care to have my son play with theirs. It’s the mother, the father was decent and friendly.

Anyway it’s been 4-5 months now since the boys last played and my son said he wants to play with the boy again and I told my partner I didn’t wanna. I said they didn’t really treat us nice well the woman, the woman hasn’t bothered to reach out I don’t want to force my son to them. It feels very off and if the woman wanted to have her son play she can reach out but never has.

Mg partner is now saying I’m ruining my sons childhood by ruining friendships and how I’m flawed in the head just like the other boys mother when I’ve been the only one dealing with them the whole time. My partner never did, I told my partner if he wanted to reach out to them and ask be my guest but I didn’t feel comfortable then he got super angry with me and just kept saying I’m ruining my son.

Am I wrong?! I want what’s best for my son and not put him in situations like that even tho the child is innocent and probably has ZERO clue but it’s the mother I feel like the mother is pressured but doesn’t want her son to be around mine. I’m not trying to ruining anything.