No Court Order & Ignorant Ex

My daughter's father and I have a verbal agreement to share custody of her. We were never married, never went to court it was just a verbal agreement that worked very well for us until about 6 months ago when he got a new girlfriend that quickly became a fiancee. I feel like she is behind all of our recent issues but that's not the point of my post. I don't know what to do. We switch Sunday at 5:30. I don't want her to go back. Yesterday, the fiancee wrote me (because according to her, I don't need to discuss my daughter with her own father 🙄) and said they were going to the river. I asked if she had a float and if not they could use mine. (She is 3 and cannot swim without one.) She replied with "Naw she doesnt need one she doesn't go out far I won't let her." My response:

Me: Well I mean she's 3 so as a safety precaution she does need one lol things happen so fast with small kids!

Fiancee: She doesn't need one though lol

M: She's 3... Yes, she does.

At this point my ex "supposedly" takes her phone

Ex: Did we ask your fucking opinion no we didn't and you are NOT gonna talk to us like that when we are trying to be nice we actually watch her unlike you so mind your own damn business when she is here

Me: She's my business no matter where she is and I just said she needs a float on. I wasn't being smart or mean.

Ex: ur a retarded dumbass mind ur own fucking business

So I get talked to like that because I said they need a float. Lol. I'm shocked honestly. It's common sense. So I technically don't have to let her go until they prove paternity and take me to court since we were never married but at this point I need help. Prior to this incident it was that I'm not allowed to check up on her during their week and they will update me if they feel like it. Prior to that they accused me of knowing about a mosquito bite and not telling them. It's always an issue, and I want to put a stop to the arguing and treating me like this. But as soon as I go to court for simply just a parenting plan they're gonna go for full custody because they are vindictive and have threatened it before. I have my own place and custody of my other 2 kids from my first marriage, I have no criminal record, I don't do any drugs I am not unfit. I'm just over this drama!!!