Am I wrong?

My dad passed away 5 years ago and today is his birthday. I am a huge daddys girl so of course today has been a rough day. This morning when I woke up my boyfriend almost instantly had an attitude. He usually goes for a walk every morning and today he said he didn't want to so I just asked why and he said "because I dont fucking want to" and thats where it all started. Hes been being mean to me all day and just yelling at me saying I dont care about him and I only care about myself. Every time I think the arguments over he finds something else to be mad and bothered about. I'll admit a few times I did yell back at him I was just overwhelmed and couldn't handle him anymore. No matter how many times I asked what was bothering him or tried to be calm and just focus on him he tried to guilt me. I told him I really just want to stop arguing and just be together because clearly today wasn't a good day for either of us and he said I was making everything about me and he said since he didn't know my dad he has nothing to say about it and he doesnt know what I want him to do. This upset me then he said I stress him out and my mental health issues are hard to handle and he said I just keep pushing him. (This isn't the first time he's done something like this. This man picks a fight with me over nothing pretty frequently the last time was 5 days ago) I told him I wish I would have just let him leave last time and then I said I didn't mean it and I am sorry but I really just don't know how much more I can take. I have a 1 and a half year old son and I dont need him thinking this is what love is. My boyfriend has nothing he has no phone no family or friends willing or able to take him in and I cant just throw him out on the streets am I wrong for feeling this way? Am I wrong for telling him I wish he would have just left. I do love him and I dont want to hurt him but this is literally ripping me to pieces