Bit long do I apologise but I could do with some advice!

Little background to our situation- I have an 8 year old from previous relationship and a 4 month old with my partner. We don’t officially live together as he has his own house that he is currently renovating (has been since before we got together 3 years ago!!) and he stays there around twice a week but is at mine the rest of the time.

Yesterday, my partner got in from work he went to have a shower and then suddenly got stroppy and then went to leave, as he was doing so he turned to me and said I need to sort my shit out, I’m bone idle and all I do is sit on the sofa all day every day... this argument went on for a while and he was throwing nasty digs saying I don’t do the housework or I do minimal and I haven’t done the washing this week and he had no clean boxers. I must admit, my house can be messy and I don’t make it a priority at the moment as my 4 month old has colic and reflux so it’s quite hard with him sometimes and other times I want to play with him and enjoy this time before I go back to work. I think he said that I sit on the sofa all the time because when he left for work at 5am I was on the sofa with my baby (this is due to my partners snoring waking us up and keeping us up several times during the night) and when he got back home (1pm) I was still there- but little did he know that I had done the washing and I had then been at football for my older son all morning and I had got back home 20 mins before him and I was getting my baby to sleep.

I ended up getting real angry and hurt about it and told him to leave and leave his key which he refused as he has a lot of stuff here... so I packed all his stuff and he’s now took it all to his.

After calming down, I now think I’ve over reacted but I also don’t want to be treated like a mug- I do everything for him, dinner, his pack up for work, washing, cleaning up after him, food shopping- the list goes on! And he explodes because he had no clean boxers, which is his own fault due to him coming and going and having bags of clothes in the house that I didn’t know what he was doing with them...

Just don’t know what to do!