Need Advise

I just called my bf and he said he couldn’t talk because his brother called him and I said ok thinking it was going to be a quick call but almost 2hrs went by and I text him if everything was ok and he finally he responded like 10min later. I called him and asked what he talked about and he got mad saying I was controlling him wanting to know everything he did. There’s part of me that still thinks about the things he did in the past and I try not to judge him but it’s difficult. Honestly he made me cry just now and I hate feeling so vulnerable. I wish he understood my point of view.