Overprotective?

As
My fiancé has always had an issue with me doing stuff without him. Anytime I mention being out of the house it's "where are you?" "Why'd you go there" he says it's curiosity but it really bothers me. Recently I've talked about going to see my brother (he lives in another state and I don't see him much) my fiancé has an issue because there's a slight chance my ex could be there. I see it as he either doesn't trust me to defend myself or doesn't trust me to resist an exs possible advances. I really just want to see my brother. Am I being unreasonable for taking this as him not trusting me? Am I right for being concerned of this progressing to an overprotecting and controlling marriage?