Don’t know what to do

So my son father and I have been separated for about a year now I allow him to come to my house to see my son when he likes which usually means he’s trying to be in my house every day . I want my son to have a relationship with his father because I never had and relationship his my father. But he is always disrespecting me or trying to start a fight . We go into a fight not so long ago that he put his knee on the back of my neck which is bed with in it self but I have a neck injury from a car accident so it made it worst and still weeks later I’m in pain. I don’t want to keep my son from his father but I don’t want to continue to deal with this abuse. He said he’s going to therapy and he hasn’t gotten physical in a while but he still arguing with me over any and everything. My mother says it’s because I’m not giving in to him and letting him live in my house I truly don’t know what it is but I’m so stressed all the time. When he’s not around I’m happy and things run smoother. How can I allow my son to still have his relationship with his father without having conflict??