Baby daddy issues

So my ex who we were never married and he cheated on me claims he has rights but mom has full rights. I'm really confused. His ex kicked me out before we figured out I was pregnant. And now that I am hes trying to say he has rights I'll be living in a different state af him. Any advice? May I also add my boyfriend now is going to help raise the kid.