Fiances friend

So my fiancé we have been together 1 year he has some friends who recently called cps on us with are now 7 week old just because we made them mad. Said we were abusing our daughter ect when anyone can say we dont even her doctor told cps she is healthy as can be and shows absolutely no signs of abuse but well my fiance will jump to hangout with them as soon as they ask even after that. When i say how uncomfortable i am with it and how i think we should cut them off he says they didnt mean it and i just need to get over it and stop controlling him just cause i asked him not to go over their. How would you guys feel like am i wrong for wanting to cut them off