Kissing the baby

So every since my daughter was born I told my husband I didn’t want anyone kissing the baby on the face or hands since it’s around flu season and even a simple cold sore could be dangerous for a baby. I don’t even kiss my own daughter because after I gave birth I got a bump on my lip for the first time and still not sure if it was a cold sore or not (which breaks my heart). I asked him to let his family know and I’ll let mine know. He agreed, well the first time going over there (they live 4 hours away), he made sure every one knew not to. But the other 2 times we’ve went they all kissed her on the face and he refused to tell them not to even after I’ve mentioned it to him.. I feel like it’s not my place since it’s his family and they are the type to get offended and give attitude about it. I’m not up tight with my LO but I do not feel comfortable with 10 different people kissing her on the face constantly when I don’t even put my lips on my own daughter. He gets mad whenever I try to bring it up and says how I’m just trying to keep them from loving on her... I try to explain that there are plenty other ways to love her... any advice? Or am I just going crazy?