Do I have a right to say no?

So recently my relationship has been a mess. He left then came back, but he's still currently living elsewhere. This drama between me and him have been going on since July 1st. Since this started I've been alone with our 1 year and 6 month old, taking care of her all by myself. Whenever he was over he didn't help with her at all. He's been "ditching" me lately when he says he'll be over to help and spend time with us both. Also would like to note he's never offered to take her and when I have asked he is always "busy". His mother has moved back into town and I have a very bad history with her, I don't like her and she doesn't like me. I won't get into all that awful drama cause most ladies would cut her off right there, which I did but she's always gotta dip in any drama her son and I are having. She keeps bothering her son and I if she could take our daughter. Of course I say no and my 'boyfriend' doesn't care what I say, he doesn't argue. She recently snapped on me saying how he shouldn't have to ask for permission to take his own daughter. For one he isn't gonna watch her his mom is and two I've been her primary caretaker while her son hasn't showed any effort what so ever. I feel like I have the right to say no. Do you ladies agree?