Need advice

So me and my partner have been together nearly 2 years, we are engaged and have a 5 week old baby girl. He’s the best dad and provider for us and most of the time is the best partner I could wish for ❤️ unfortunately the other time he makes me feel so down and like I shouldn’t be with him. He’s very close with his older brother and they like to smoke weed together and “catch up”. His brother only lives down the road to us and whenever my partner has a day off work the night before he will go down there without fail rather than spending time with me when our daughter is in bed. And he will stay there for ages, till about 11pm. Obviously I’m going to bed then so I can be ready to wake for our daughter in the night. I’ve tried to talk to him about this but he just doesn’t seem to listen to me or it just doesn’t go in. It feels like I’m in a 3 way relationship with his brother and him and I don’t know how much linger I can put up with it. I don’t want to leave him because like I say most of the fine he is brilliant and I’m really in love with him but this is really effecting my mental health I feel really lonely and down and like I’m pushed to the back when he does this. It’s not like he doesn’t get to have his free time he gets to meet up with his friends etc but I never get to do anything I want to do and have free time. Eugh I just don’t know what to do.