Struggling relationship

I've been with my boyfriend for about 5 months. Due to a bad past I really struggle with trusting people, especially men. Recently he got drunk and said some things he shouldn't have, "i still love my ex" "I am using you to fill the hole". He apologized and said it was wrong and I really don't wanna break up with him cause I love him. Last night I got asked out and I turned the man down and told my boyfriend and he demanded to know who it was. I got scared and didn't want to tell him and then he got really mad. I dont know what to do to fix this