Not sure if I’m being a psycho

So I’m engaged to a great guy overall. We’ve been together 4 years and we have a 1 year old together. Overall we have a good relationship but since having the baby things have changed obviously. He’s a great dad very helpful and helps around the house. So we always come back to this fight. I don’t go out much I’m content being home. We live in the country 2 hours away from where I’m from. We have a group of friends we hang out with but we all have kids and all that so we don’t get together as oftenAnymore maybe like once a month. And I work as a nurse. Anyways the fight we constantly have is my fiancé goes to play hockey once a week and then works in the day and cares for our daughter while I’m working. He often goes out with his friends or has people over 2-3 times a week. And they’re people I don’t really like. He has this one friend who has been over constantly lately like I swear every day this week. and I’m so pissed because it seems like he can never turn this friend away to spend time with his family. And he doesn’t get I don’t want to see his friends after I get home for work multiple times a week or I don’t want them over when I’m trying to sleep for my night shifts. Am I being insane. We just got into a big argument about this. He says he turns down offers to do things all the time and I shouldn’t be giving him a hard time because he is an involved parent and it’s not like we never see each other I mean we live together. But it just bothers me that he doesn’t get why I think hanging with his friends 3 times a week minimum when we have a toddler is annoying. Am I wrong here. Why can’t he just be content to hang with his family and see his friends 2 times a week. I’m just very annoyed.