Could he take the kid away from me?

So I'm about 8 weeks pregnant from a married guy. It's a long story I know I messed up and I'm a terrible person but I don't need all the judgment right now. I'm over that. I decided to keep my child, and told my parents who are being very supportive even though they're really upset. My parents don't know a married man got me pregnant though. I had to make something up saying it was someone from school who vanished after I told him I'm pregnant. I'll take care of that part later, I just think dropping the bomb on the right now would be too much. Anyways, I told the man who got me pregnant and he suggested abortion, I said no. So he said he can't tell his wife about me & him plus the kid. I said that's ok, not asking for that. I said I'm just letting him know since he's got the right to. We don't see each other anymore although he wants to, I can't do that anymore. He texts me occasionally to ask about the baby and me. Recently he suggested me to give the baby up for adoption, so him & his wife could take my baby and raise him or her where he thinks would be better to have a mom & dad around. (His wife can't have kids &a they've been talking about adoption for years so he told me he would be able to convince her in time to adopt my baby) But I said HELL NO. I want to raise my kid w/ my parents where he or she will have their real mom around. He said ok. So that means he doesn't want the child the way I do. I would be ok if he told his wife the TRUTH and wanted to be part of my kid's life but he said he can't. It's either adoption where his wife wouldn't know the kid is his & mine or he isn't going to be part of the baby's life. It's a shame bc I wish my kid had a father around I know how it makes a difference in a kid's life. But my concern here is that what if in the future he decides that he wants the kid, would he be able to do something? Would I be in trouble? I'm really scared of that.