Do you ignore a 20 month old?

DD is very independent. she plays by herself and keeps herself entertain but she also want someone to play with like any other toddler correct? My thing is my bf yelled at me today because I told him I wanted some me time. my "me time" consists of getting coffee and doing some crochet which I barely got into. he says he doesn't understand why I can't do it in front of the kids that to HIM he doesn't need "me" time and he would do whatever he wants around the kids the only thing he can't do is smoke weed around them so that's his time when he goes out to his friends house. I work full time and he currently unemployed and we got into a really heated argument today. it made me cry & I explained to him that he ignores DD when she tries playing with him and he's too busy watching videos. he expects me to do the same and I just can't do that. He says I need to "fucken stop babying her" and I don't think he understands how I look at things. if I was dishes or clean around she does play by herself or I tell her nicely to give me a few minutes. am I wrong about the whole thing?? I feel like she is still little and learning to just leave her and ignore her all the time.