In law issue

My husband and I have been together for 8 years. When we first started dating he opened up and told me that his Dad physically abused him as a child.

Every once in a while, he will bring up the topic, or tell me a story of a time when he was abused. His Mom generally knew what was happening and didn't do anything about it.

I have always felt bad/negative vibes when I'm around my father in law (maybe it's just because of the stories I had heard of him) And it often feels like he's trying to provoke me by bringing up topics that are controversial (he's homophobic, racist, sexist)

Now we have a 1.5 year old son, and the grandparents want to take him overnight sometime. I told my husband I didn't think I was comfortable with them taking him for the night, and my husband is telling me that I'm "being irrational", that he wouldn't abuse a 1.5 year old, that we don't know if he has changed, and shouldn't he be be given another chance (the two of them have never spoken about the abuse, he's never apologized or shown regret)

Am I being irrational? I know it's not likely to happen. But????

I think what bothers me most of all is that my husband isn't more uncomfortable with the situation. I feel like he just wants to avoid a potentially awkward or confrontational situation.