Thinking of cutting my sons father out of his life

My son is almost 7 and his biological dad has never been around. The man that has is my ex who has been around since I was pregnant. We were together for a long time and I left him 2 years ago because of his cheating and abusive behavior. He pushed me a few times and was very verbally abusive. He has never been abusive to my son and never pushed me in front of him when we were together but hes still mean to me in front of him when we have to see eachother. My son keeps asking me lately "what if dad hurts you?" It breaks my heart. His dad is very bitter toward me for ending things and for moving on. Whenever we have to interact he's always so damn short and just mean. I don't want my son to see that anymore. How do I deal with this? Should I stop letting him see my son?