Is this acceptable to you all?

Jennifer
I'm just not sure if I'm crazy feeling this way or if my right. Please let me know what you think.
So, hubs and I have been together for 8 years, married for almost 3, have a 2 year old little girl and are TTC with #2.
He has a friend from work that he's know for years and this friend has recently separated from his wife and is in the middle of a divorce. I feel bad for him because they've been married forever and he didn't want the divorce at all. When he moved out of his house with his family, he moved into a  house that his mother owned near our neighborhood.
The first week he moved in, my hubs wanted to go over there in the evenings a bunch to keep him company since the guy was feeling down. He'd wait until after our daughter went to bed then head over. 
This was a couple months ago and he still is going over to his friends house like at least 3 or 4 nights a week! He'll usually spend a little time with me after baby goes to bed, but then I end up heading to our bedroom alone and he leaves. He usually doesn't come home until way late, like last night I woke up to use the rest room at 2:30 and he still wasn't home. 
I don't worry about him cheating on me, but it's starting to get old going to bed alone (especially when we're TTC!). And I just feel safer when he's there with us at night. 
He also works weekends so his 2 days off are during the week and he's home with our daughter those days. He sleep so hard and is hard to wake up in the mornings. I worry that when he's home alone with her, he's sleeping half the day.
I want to talk to him about this, but I'm just curious what you all think. Am I crazy?!