Dealing with ppl you don't like around your baby

So the holidays are coming up and I'm sure I'm not alone here when certain relatives or ppl you try to avoid seeing any other time of the year is getting unavoidable now.

Finally had to bite the bullet and visit my mother in law when her boyfriend was there. He's a Vietnam vet and well basically is not the most respectful person and lives off my mother in laws money. I went to go get a cough drop from the car and came back and my little angel was on his lap. It made me cringe a little. He's not a horrible person but he's just not someone I feel comfortable around my baby. But my husband says I need to get used to it since it's his mom's boyfriend.

How do you ladies deal/cope with these situations? It's my first baby and I am definitely mama bear when it comes to her but don't want to be made out as an evil bitch who won't let certain people around her baby- especially in a situation where I don't have much of a say.