Am I right for not letting my son be alone with my mom and her boyfriend?

Long story short my mom knew I was being molested by my grandfather when I was little but didnt do anything about it. Never kept me away from my grandparent's house. Anyway fast forward my son is almost 3 year old now. We are on good terms and she was there for my son's birth, and has been there for me and babysat my son before alone. I trust her. Well just maybe 3 weeks ago I found out that she has a boyfriend living with her and he's fresh out of jail for evading arrest or something... she said it wasnt anything bad that he did. Well, I told her that I don't feel comfortable with my son there without me even if she is there just because of what she let happen to me. Im not going to be like her and just put my child in a potentially dangerous situation. I dont know this guy. Whos to say he wouldnt do anything to my child? Anyway, she said that shes happy and that i dont want her to be happy blah blah blah because I wont let my son go over there by himself anymore. I told her i can meet her at the park or she can come to my house and visit with my son. She still throws a fit and shows up wanting to take my son to her house. I tell her that he's not going to be alone over there. Im not taking the chance. Now she doesnt call me anymore and never invites me and my son to do anything with her. We were beginning to be really close. Now we arent because she is mad at me. Can you ladies please tell me if I am right or wrong? What would you do?