is it just me or is it men.

My boyfriend has a serious problem of not coming home when he says he is. 
I trust him it's nothing like that because I know where he is, just the fact that he says he'll be home at this time. 2 hours later he rolls in. 
The other day he was at his mums and he rang me saying "he's just doing something then he'll be leaving" an hour later he comes in. Bearing in mind he lives 5 mins down the road. 
If I say something he's just like your over reacting. I just sat and spoke to my mum. Then tried putting the "I was with my family" act on me. I trust him 100% so it's nothing like that. Just I hate it when he says one thing and does another. 
Am I in my right to kick off? Or is it just men????