Ex husband still tries to manipulate me

I left my husband because he was abusive, physically, emotionally, mentally and it was starting to become more and more frequent. He is also a drug addict with schizophrenia, which made it worse. We share one child, and he is almost 20 years older than me. When I left him I got a protection order against him and he didn't see our daughter for close to a year.

When he contacted me earlier this year I let him start to see our daughter, for small amounts of times, then day visits gradually working my way towards overnight. Last weekend he came to pick her up and it wasn't until he called me later that night to tell me she was asleep and he will bring her home in the morning that I realised he was high. I tried not to worry as I knew she would be safe as he was at his mothers house but his mother called me the next morning saying she was concerned for my daughter and I need to get her home immediately. So I got her home, and I told him that he won't be having her again without a drug test.

He has since texted me saying he's going to take me to court because I'm running an illegal escort agency (because I help my sister with her massage business admin work, and I used to be a SW) that I'm an unfit parent because of my weight( which I'm actively working on) because I apparently do the bare minimum for our daughter (I have her everyday, while he has her once a fortnight, sometimes twice when he's not hungover or high). He also accuses me of palming my child off to every Tom, dick and Harry, while I do drugs and prostitute myself.

I know he has no grounds to stand on but what annoys me is he will threaten me of all this stuff for shitty little reasons like I blocked him on Facebook (because he was sending nasty messages) or because I made plans for Christmas Day. Just petty reasons he will threaten me and I have to argue my way out of it. I just wish we could get along, I try my hardest to be honest, with an open line of communication. I don't expect anything from him, not even to have his child every single weekend. I don't start the conflict, he starts random childish dramas for nothing. I really don't know how to deal. Every boundary I place he tries to tear it down it's starting to get exhausting.