Boyfriend smokes weed and plays video games all day

My boyfriend and I have been together almost 9 years. We have two kids together 5 months and almost a 2 in a half year old. All he does most of the time is hang out upstairs in his “man cave” and smoke weed and play video games.. probably sees his kids maybe 20 minutes total in a day? I have nothing against weed because I’ve smoked it and eventually want to get my medical card.. but he doesn’t do much with the kids besides give our daughter a bath in the evenings. He’s also unemployed because he walked out on his job of 10 years (long story) I just think it’s a little ridiculous how I take care of our kids all day and he’s basically never around. I’ve tried talking to him and it just turns into a argument every time. I just don’t think it’s fair how I’m taking care of our kids most of the time and dealing with chaos and constantly stressed out yet he’s in his own little world getting high and playing video games almost 24/7. Am I being irrational here? I’m really starting to get fed up with it. At this point if anybody needs to be smoking weed it should be me from what I have to deal with all day lol. I just want others opinions.. I honestly don’t think I’m 100 per event in the wrong here with my feelings,