Paranoid nanny

I was sexually abused as a child (keep that in mind before I say anything more)
I LOVE children, you know how some people are just naturally good at things and "destined" to do something in life? Well for me I think that's being a mother. I'm so maternal I always have been everything baby just excites me. 
My current girlfriend is the exact opposite she's more of a realist and she keeps me grounded in a sense that she always reminds me of he financial and she doesn't want to have a baby until she's 150% ready and able to support herself, the baby and her partner to the best of her abilities. 
So because I have her being a realist she suggested I work with children and I am now a nanny to the cutest little girl ever (almost one). She loves me, I'm able to keep her calm, she behaves and responds very well to me, she looks up to me and she learns from me (I've taught her how to count to 10, she knows my cat by name, she sings with me — she's just very responsive.) 
I've started getting at that stage where I'm very protective over her for example when we're out and about sometimes people will come over and just touch her without permission and it really pisses me off like get your gross stranger hands off of the child learn respect. 
Anyways so the women I am nannying for her sister had some issues with paying her rent or something and she lost her home and has moved in with the women. 
Her sister, her sisters boyfriend, and all of their animals are in the basement of her home. Where I stay. 
So I was told that they would be no issue they won't come bother me etc etc SO the man decided one morning that he would loom over me and the baby. Every where we went in the house he decided to go. 
He was playing horrible music out of his phone on a very high volume and just trailing behind me. 
He was being very... I don't know. I pick up on certain vibes and when I do I get very anxious around big men (big meaning significantly  larger than I). So yeah I got that feeling and I was trying to bring the baby away because it felt off. He then proceeded to ask me personal questions about myself one of which included my age and when I told him he cracked a joke about "barely legal" females and babysitting in "certain films" referring to porn. It was just so uncomfortable.  
He told me I'd make a great mother and asked if I had a boyfriend I said no I have a girlfriend and he said "oh well you'll need a man someday" *gags* at this point the baby had made a BM and I could smell it and just felt uncomfortable changing her in front of him because he was just making weird comments about her as well and kissing her but not on the face or in normal "awe baby" ways like it just felt uncomfortable so I was trying to keep her out of her general area. He ended up walking to another room and I figured nows a good time. 
So she's a fussy baby when it comes to changing a she doesn't like it at all and normally throws fits so I like toprepare her and distract her before I change her and I show her the new diaper put her on the surface (sitting) and ask her to lay down 9 times out of 10 she does and then I make it a game and pretend to eat her toes before I change her so that she's happy so this guy must've heard me do that because as I was wiping her he was creeping around the corner when I say creeping I mean he was trying to not be seen, peeking, creeping behind the corner. 
Idk I don't want to accuse this man of being a peadofile or a rapist or anything just because my past may make me more paranoid with men and babies (It 100% affects me on a day to day basis where I have a real fear of men and being alone with them especially if they are larger than me and I feel like I'd have trouble defending myself) but I'm not sure if I should say something to the parent or just keep it to myself and keep a real close eye on the baby and wait until I have more to go against because I am with the baby from the time she wakes up till the time she goes to bed (she sleeps in the bed with her mother) so if anything were to happen it'd be on my time. 
This man could just think he's being an outgoing comedian and not realize I find him insanely creepy. 
A LITTLE UPDATE: the guy is a known meth addict and the reason they lost their house is drug/ money related I just found this out today. I talked to my mum about it and she helped me work a plan around it. Whenever I feel uncomfortable or in a compromising position I leave right away and keep a pre packed diaper back near the front entrance of the house so I can make a get away and I bring the baby over to my mums house OR I bring her out to the park/mall. Basically anywhere but there. Thank you so much to all the positive advice you ladies are wonderful ❤️