Thinking about leaving

My bf and I have hit a rough point and it just seems to be going down hill fast. I’m not happy and he doesn’t seem to be either. We live together and have a 4 month old together. We used to tell each other all the time that we loved each other and kiss and hug when leaving but now. Nothing. Especially when something doesn’t go his way like I forgot to put his energy drink in the fridge so he didn’t have a cold one for work or couldn’t make him breakfast because we didn’t have anything to make for breakfast this morning (going grocery shopping today) so he gets an attitude and refuses to kiss me or show any kind of affection. Gives me silent treatment. If I have him hold the baby for me to do something (usually FOR HIM) and I take too long or won’t come get the baby exactly when he wants me too he tells her that I don’t want to hold her. I don’t want her and stuff like that. I’m not happy. The apartment is in my name and I lived here when we got together and he ended up moving in with me but I’m ready to just pack our stuff and leave. He can keep it or move out, whatever at this point. I love him but I’m just not sure it’s worth trying to work things out. What do y’all think? Work things out or just leave?