Losing friends with age

Is it a normal thing to lose friends as you age? I feel like I had a friend group when I was in high school. Now I'm five years out, finished college, and I only have my boyfriend (who was once my best friend), my best girl friend, my sister and my coworkers. I had a little group of girlfriends not very long ago and it just didn't work out. Starting to think that I'm the problem which sucks because I have good reasoning to not be friends with certain people. I'm just over negativity and toxicity and unneeded drama in my life, but I still feel lonely sometimes. I'm very extroverted so it's weird for me to just have a close circle. What about everyone else?