Am I withhold him from his kids?

My ex loves to say I only let him see or get the kids when it’s convenient for me.

I’ve tried to get him to take them on weekends or ever other weekends. But he told me his mom couldn’t handle 3 kids, on the weekends. Because she goes and get his daughter every weekend.. then he wanted to get them two days out of the week. I said ok. But asked him what about school, he wanted to pick them up around 6 pm but then bring them back to me. So I could get them ready for school and take them, and pick them up from school and do all the sport or other activities😂 yeah no. I told him he would be responsible for doing all that. He didn’t like that. so idk. Majority of the time, he ask to come over here to see them. He has tried to invite himself over for dinner many times, or he will text and ask to see them after 8 pm. He has even asked to stop by see them at 6 am before school. I say no to that cause, I don’t think I’m obligated to have him in my home.

Are kids are 5 and 6 years old. They are in bed by 8 pm and he never ask to pick them up before school to take them to breakfast and to school. He just wants to stop by and get them upset before school.

I also want to say he isn’t asking to see them every day or week. Maybe once or twice a month, sometimes we will go a whole month without hearing from him! 

We got in to a argument the other day cause my dad was hurt and I ask for him to keep the boys. He did keep them that night. Well his mom did… But he wanted to bring them home at 8 am. My eye was swollen and my head hurt cause I just lost my dad. I ask for him to wait till later that day so I could sleep and grieve. He said no. Then went on this whole thing about how I only let him see them when it’s convenient for me. So he wasn’t going to. So I told him I’m tired of this, maybe it’s time to go to court. He got upset and said I was threatening him? Idk how going to court to get him some visitations is threatening.

What am I supposed to do. Start letting him come and visit here. Cancel our plans for him. What if he don’t show up, cause that has happened lot! Like today. He was supposed to pick them up and take them to an easter egg hunt.. he ignored all calls and text. Idk but this is stressful!!