Child support and withholding child?

I have an almost 6 year old with my ex, we separated before she was 2. I’m her primary caregiver, he picks her up 80% of the time for weekends. Since our separation I’ve never gone for child support. Mostly out of guilt, he always has some reason why he can’t pay or contribute to anything outside of his time with her. There’s been times where I have gone to him and asked for help, whether it was pitching on extra curricular activities, school uniforms, lunch food, trips ect.. he always refuses and says he can’t afford to help, then I’ll hear about his vacations (mostly minor stuff like camping for a week, cottage weekend ect) he gets to take her to do all the fun stuff when he has her and he has the ability to do those things because he doesn’t support her or pay for all the essentials… and sometimes it bugs me. I want to take her to have fun and do activities like indoor water parks, camping trips but I can’t afford it and I know it’s my responsibility to be able to provide for my child but I was a SAHM mom up until our separation, I had no income from a job. No money of my own. Nothing, when we separated he refused to help with child care (even watching her himself so I could work) he took everything form me, all my furniture, everything I owned and we owned together, the money and left me to struggle… I worked two jobs, I paid for her child care, I started fresh and at one point I lived in a shelter. I fought hard to get where we are now and have a stable home, food in our tummies, clothes on our backs but I live paycheck to paycheck and it’s hard. I applied for legal aid, it’s a long process before I’d even get to speak to a lawyer and start a custody hearing… I asked him for $20 to pay for a trip and nothing, no response for days and then him asking when I can drop her off to him? I am at the point where I just want to ignore him completely and keep her home. I feel horrible for feeling this way, but we both wanted a child and unfortunately our relationship didn’t work out but we both should support and play roles finically and physically in her life, it shouldn’t be all on me? I guess I’m just venting