Advice please!

me and my boyfriend have been together just over s year, we have a 18 week old baby. We don't live together as he cares for his daughter in the week, he comes down most weekends. the problem is the ex. his daughter's mother, she was supposed to have their daughter back full time after I had the baby to help us out but that hasn't happened. that's not really a problem but my boyfriend does what she says and sometimes keeps his daughter weeks at a time without seeing me and our baby. he doesn't have the baby at his house as his daughter hates noise and needs a lot of attention. I understand that and I'm okay with it but I am left to care for our baby alone (I also have a daughter from a previous relationship) he said he would be there but they are empty promises and he cannot do anything as his daughter's mum is next to useless. I understand she needs him but I also need him too. I'm really not sure what to do, I feel l9nely and sad when he's not around and feel like a single mum. again. I'm not sure if I'm being silly but all I ask is for some more time with him yet he doesn't understand where I'm coming from. I love the bones off of him, he's the best dad to his daughter and our baby and the best boyfriend. lately I'm thinking I'd be better off alone to save my feelings. has anybody any advice please?! sorry for the super long post!