Is this controlling?

On some my days off I go to another town that I used to live in for dinner. A relative from that area passed away so it feels peaceful sometimes. I don’t spend all day there and don’t even go everyy week. But my fwb doesn’t like it and comments about it all the time. Earlier he made a joke and said “if you’re not sleeping I’m gonna come to your house and beat you up” and then was like “you’re probably just in (town)”. I said no you’re just being paranojd bc you don’t like that I go. And he said he doesn’t like it bc I “lack communication and am secretive” about it because I don’t tell him when I go and what I’m doing. Also admitted he gets worried I’m sleeping with someone there. But I told him before what I do while I’m there generally and I don’t feel like I *need* to update him on where I go and what I do. If it comes up or he asks, I don’t mind. But I don’t feel like I should have to interrupt convo to tell him “I’m going here to do this”. I told him if he’s worried that I’d let him check my phone, but he said that doesn’t count bc it’s easy to delete/hide things. I also told him I’d share my location. But now that I’m really thinking, I don’t want any of those things. This is too much for someone who “doesn’t want a relationship”.