Can they even do that?

I'm not the best at summarizing but I'll try.

So my ex husband and I have a 10 year old daughter. We divorced when she was 3 because he cheated on me. Before the divorce was final, some of HIS friends and family came to my work to tell me not to let my daughter near him, that he was on meth. I told my attorney to ask what we should do about it because I want my daughter to be safe and not around anything on his visitation days. She put in the papers that he needed to pass a drug test to get visitation because of the drug use comments. Mind you during this time, he would not talk to me. He wouldn't call me to discuss the parenting plan or anything. When our court date was set I called his parents and friends to let him know when the court date was so he could be there. When the day came, he didn't show up. I was granted the divorce by default and in my paperwork it says she is with me 365 and him 0. He stayed absent from her life, no texts phone calls etc, until 3 years ago. He got a new girlfriend who bribed him to be in her life by buying him a truck. I'm too nice for anyone's good so when they contacted me 3 years ago wanting to come by and see her I asked my daughter if she wanted to see him (she remembers him and I had old photo albums she got into of them together once) and she said yes. I told her it was her decision. I allowed them to come to our house and see her but that they weren't allowed to go anywhere with her. His girlfriend tried to act like she was going to force him to be involved and then would flip around and say how much he really does love her. 🙄 Anyway, for a year they came over, they called consistently, bought her some things she wanted/needed for school, even went on a field trip with her that I allowed, and then moved 2 hours away out of the blue. Didn't tell us, didn't say bye to her, just left. Fast forward to mother's day with no calls, texts, visits for over a year, they ask to see her. I said, you're more than welcome to come by next time you're in town, but we had plans today. So of course now the girlfriend is telling everyone I refused to let him see his daughter, when I did but I didn't, and they refuse to see that I'm being nice. My decree states I don't have to do any of this. Since I said no on mother's day, now they're trying to take me to court and get custody of her. I feel like they are just mad and talking, but who knows.