Controlling boyfriend

I work out at an all girls gym and recently have been thinking going to a different gym because the hours are better making me have more time to workout. My bf said “I’ll be mad at you” kind of jokingly when I told him I might be switching gyms to one where there is guys. I kind of understand his concern but at the same time I seemed possessive for him and he can’t really tell me which gym to go to. Thoughts on this?