Controlling behavior?

My boyfriend is 21 years old. I'm 22 years old. We have been dating for almost 4 years. My boyfriend is a bit immature while I'm more mature. We live together. Lately, he keeps saying I'm controlling him. It's upsetting. His reasons for it is because I want to know who he is with and where he is. His friends are his age or a year younger than him. They can't hold onto any girlfriend. He has been hanging around them more and more and it seems to be creating distance between us because I think he gets jealous of them and wants to be able to do whatever he wants. I don't feel like I'm controlling since we live together and have been dating for so long. We are in a committed relationship and have spoken about getting married one day - just not any day soon. Am I being controlling and overbearing? He doesn't tell me who he is with or even where he is anymore and I don't think it's right.