Custody agreements.

So my son will be 3 next month. His biological father has been in and out of his life since the day he was born. He was very emotionally/borderline physically abusive and I left when my son was around 4 months. His dad, after failing to get me to come back, moved 4 hours away to another state and wanted to have 50/50 custody. I wasn't comfortable with it because my son was under a year old and felt strongly that driving 4 hours one way every two weeks was too much for a baby and he needed to be in a consistent environment. On top of his insurance only being through me in the state I lived in. His dad threw a fit and didn't ask about him or see him for a good 6 months if not longer(I cant really remember how long). When he would come back to see his family, I always offered to meet him somewhere so he could see his son, but wasn't comfortable with him taking him due to the fact his family wasn't even involved with my son and my son doesn't really know any of them, but he never wanted to always stating how I'm selfish etc. Fast forward to now, he has moved back into the same state now living an hour away from us. He lives by my family, so every time I go see them, I offer for him to meet us and see him and get to know him. Even though he never asks about his son and doesnt know anything about him.. I've explained how it's not fair to put a toddler in an uncomfortable situation and he(my son) depends on me whose been here the entire time to keep him feeling safe. He met us one time the past month and picked my son up and my son started freaking out so my mom tried to take him and his dad wouldn't let him go causing my son to hit him and cry. Got home that night and noticed fingerprint bruises around his thighs.. I called him and made him aware of it and of course he denies it being from him, but he's always been the type to never take blame for his wrong doings. He's threatened so many times to take me to court and I've always said go for it because I truly believe a judge will do what's in the best interest for our son, not either of us. I want him to be able to have a relationship with him, I've just tried to explain when you abandon a child at a young age, you can't just jump back in like nothing happened. It takes time and meeting up to see him a few hours was in the best interest of our child so he feels secure and has an out if hes not comfortable. He refuses to work with that though and states pretty much every day how I'm selfish and this is why so many dads cant be dads ect. My question is.. Am I doing this wrong? I became a mom at 17 and now being 20 I know I'm still young and I can admit I make mistakes when it comes to parenting. I know I don't do everything right, I just want my son to feel safe. He has tried taking him before and my son just freaked out screaming for me. Just a little more info, he is on child support and 6 months behind, plus never bought our son any presents for bday or Christmas except one time but he took the toy with him instead. I never let that interfere with him still seeing him.