Is it a bad idea to date a guy that was in prison?

My boyfriend and I have been in a relationship for two years. We met on POF and things are going well. We met when he still had his ankle bracelet on. I knew the reason why he was in prison so that didn’t scare me. He had a gun in his house that was unregistered at the moment and other things has to do with weed, no killing people or anything crazy like that. He is a good person, he’s there for me and my son (not his kid) and in general he treats us well. I met his family and friends, they seem mature and wise and my boyfriend gets along with them well. What bothers me is who ever finds out that I’m dating a guy who’s been in prison, they freak out and say I’m stupid in love with him that I ignore that fact. Am I crazy for being with him or are they ignorant?