I'm hiding my son from his bio dad?

Emily • 28. Married. Mama.
Ok so, my son's biological father hasn't seen him or even asked about him in 2 1/2 years. His bio father is in a relationship with my husband's cousin, and they just had a child together. Since then, some things happened and that part of his family hates me now. Anyway, he's there for their baby and her other child(which could possibly be my stepbrother's child). It's good that he's there and taking care of my husband's cousin's children, I could care less. Since they've been together, him and his family has told her family that I'm keeping my son away from him and I won't allow them to see him. They also think because my stepfather has some money(definitely not rich but has more money than they do) that we think we are better than them, which I do not. Long story short, we have his family and my husbands family blocked on social media so we don't have to deal with drama. But we heard from one of his other family members that they've posted  something about me "hiding my child from his father". How am I in the wrong? When he's had multiple chances to be a dad to my child and blew them all, then doesn't even ask how he is or asks to see him or even acknowledges him when we seen them in public and has to the nerve to tell everyone I'm the bad guy.