Am I being annoying/difficult?

My kids are 3 & 4. Their dad left when my youngest was barely 2 months old. He was manipulative, controlling, an uninvolved dad, shitty person. He wanted to go out, find girls, etc. Disappeared for 3 years. I reached out to him in November and tried to get him to come back around. He seemed okay with it until it came time to actually visit and then he didn’t even reach out and stopped talking to me. But then I got court papers where he says he wants them on weekends. Throughout the court proceedings, where all I said was that I wanted a more gradual introduction, he unnecessarily lied, saying I kept them away from him, that the restraining order we had was a lie, that our relationship made him smoke weed. He just started getting visits a couple weeks ago. As soon as I dropped them off at his first visit, he said he needed to change days and asked me to go outside of the order. I told him he needed to change it. Even tho he just picked it 3 days prior. He said (LAST WEEK) that he would. Haven’t heard anything about it. Even though he said he’s at risk of already missing a visit next week. He also warned me that he knows he’ll probably miss at least 4 other visits too. And a week of no visits next month. Now him and his aunt (who HE PICKED) keep trying to push this idea of “being flexible”. But I don’t really want to “be flexible” right now. I think it’s funny he all of a sudden wants to work together now that he knows he’ll be missing a bit. He’s 2 hours a visit 2x a week and then courts will review if he showed up and did good with them to decide on more time. If he can’t handle that, I don’t want to enable him getting more time.