Custody battle.

Me and sons father had mediation. He didn’t want to agree to me doing more visits with him and I didn’t agree to our son doing overnights until he’s a bit older. Now my ex is accusing me of having sex and spending the night late at night at his house. He also says he has multiple witnesses to this and that his roommates are gonna tell the judge it’s true,and that because I APPARENTLY SLEEP WITH HIM, this will be a way to loose sole legal custody. I have sole legal custody now. I’m afraid since he has witnesses and that I’ll loose my son. Can this happen in a custody battle? ☹️😞 I cant eat sleep nothing. Can’t even go to work.