Need help and advice

Sorry if this is in the wrong group.

From the age of 14 to now (18) my brother has tried touching me inappropriately and making inappropriate comments/questions (such as "do you shave down there?" "Are you a virgin?" Etc.). I'm trying not to see him as much to avoid it but I told my mum the other week and she said she'd talk to him but I need to start living at home more (I am supposed to live with my dad and brother and see my mum occasionally but since the events I am currently living with my boyfriend's family).

Anyway recently I've been remembering the incidents of him touching me and getting really angry and easily annoyed, my boyfriend feels like I'm taking it out on him and doesn't know how to help.

Does anyone have any suggestions on how I could stop taking my anger towards my brother out on my boyfriend or how I could move past these memories?