Boyfriend doesn't want to visit my hometown

I have an issue. My boyfriend is black and I'm white. We're both 17 but I'm emancipated. We live in Springfield Illinois. Well I asked him if he wanted to drive with me to my hometown (Anna, Illinois) and he said no. I asked him why and he said because that town is racist. I said no we arent, just because there's no black people there doesn't make it racist but he declined anyways. So I asked him what I could do to make him stop being so wimpy and he told me he just doesn't feel comfortable. He told me he drove through there once and there were too many Confederate flags. I told him that doesn't make you racist, it's just like an American flag or a Canadian flag. I'm really upset by this because I want him to meet my family but it just seems like he doesn't care.