Is it ok to have pets

Is it ok to have pets around a newborn it’s just I had two kittens at 16 weeks old and two guinea pigs at 9 months old and the social workers kept having an issue with the kittens saying they were always around the newborn and made me get rid of them I have had them from before I got pregnant and I am a bit upset about them going but am I right in saying u can have pets around a newborn if they are supervised