Am I being selfish?

Clarissa
Ever since my sons "father" decided to no longer be in our lives I did try and talk to him to be sure it was what he wanted he LOVES to change his mind like if it's okay to hurt me and baby then come back like nothing happened. Anyhow it's been about a month that I tried to knock some sense into him and try to have a decent conversation to see if it's actually what he wanted a 100% sure kind of thing although I knew he wasn't. I have gone through some things since then and I feel OVERLY PROTECTIVE of my unborn son. I'm at the point that I COULD CARELESS IF HES THERE OR NOT. I feel better if he just stays away. The reason I feel this way is because I have a 3 year old son with someone else and my God he's so horrible to me and loves to make my life hell bringing our son in the middle without realizing how he's affecting our son. 
My point is I don't want my unborn sons dad to use our son as a weapon to hurt me or whatever cause at the end of the day it's my kids that are being affected and they're being assholes about it. I know sooner or later he's gonna want to come around because he's gonna feel like playing "daddy" role but I DONT WANT MY UNBORN TO GO THROUGH what my 3 year old is going through being put in the middle of mom and dad.
Does that make me a horrible person for wanting my unborn with me and me only at all times ? :(