When do I talk to my guy about my concerns?

We are really into each other (had a few dates and had sex), but it's only been officially a week. We both want to be in a relationship but I have some reservation. I am afraid of how my family will react to us. I am a white, 23 year old female; He is black and 31 years old. My mom knows and she isn't happy about it but apologized for being racist. She still doesn't like the idea of him being black and the fact that he is eight years older and is concerned. I am super scared of my brother finding out, I would not be surprised if he got threatening and violent. I really want this to work out. Do I keep this from the rest of my family for now? How/When do I talk to my guy about my concerns (racist family & 8 year age gap)?