Anxiety going to ex’s house

So my ex/Bd and I broke up bc he cheated on me and is now with the girl he cheated on me with , we had sex a couple of times since our 6 month old was born . We both confessed that we still have feelings for each other but he doesn’t want to pursue anything bc he has a gf who is now pregnant . So I’ve really try avoiding him and only try to talk to him about our daughter. But now he’s parents are always asking why I never come over anymore bc they wants to see there grandchild but he gets mad when I drop her off so I would have to stay there ., and I really hate going over there bc I get really bad anxiety and makes me super depressed . I don’t really know how to tell them , I would just tell them I can meet them somewhere but they’re going to know why and it’s not really they’re business.