I don't want to be responsible for him and his kids

I'm sorry if I sound like a horrible person here.

I am 24 and a single mum to a 10 month old. Her dad was never in the picture and I'm fine with that.

This other guy who I have known for maybe 6-7 years has been a good friend of mine especially in the past 2-3 years we have gotten closer.

He recently separated from his wife (June 2019) and in her rage and spite she doesn't see their kids. Her choice right? Can't force a parent to be there if they don't want to be. I know he is fully keen on giving her access to the kids if she wants. But she always says no. She's busy. She won't respond. Etc. He has been keeping track of this so when the time comes to apply for divorce, he also will fight for sole custody since again he has had all 4 of his kids since June 2019.

He has recently admitted his interest in me. And I made it clear that I care about him as a good friend but I would never spend that kind of time with a married man. I'm not interested but perhaps once his divorce is final we can see where we are in life. He seemed ok with that answer for a little bit. But now? He's determined to be with me. Professes his love all the time. Buys me coffee all the time, lunch, little trinkets, a necklace at Christmas. I have no problem being his friend and I really tried to draw the line at nothing more than coffee type of thing. Doesn't listen. He isnt stalking me let me make that clear. I don't feel threatened by him or anything. If anything I feel sorry for him because I know he's lonely and struggling and needs support more than ever.

His new kick now is looking at houses for rent. And he'll send me links and I'll be like oh I love that colour or no that kitchen needs help and all that. There was no issue until he started saying "WE". We should look at this house. We could afford this. We need that. WE.

Very bluntly I told him I can't live with him especially knowing how he feels about me. I'm too afraid to get close to someone who is married because we have all seen what can happen there right?

He wouldn't listen. So I said I don't want to be responsible for all of your kids. I have one child. She is small. She is an easy baby. I make enough to support the two of us. She is my world. By no means do I want to support 4 other children. They are great kids. But kids are a lot of work. It would be this instant family and financially I can't do it. Sure he makes money himself. But I don't want to be dragged down with him and his debt and the kids sports and the big vehicles and big house and all the laundry. I'm not cut out for that. And I told him this. Because that's a lot to ask someone who you aren't even with. He called me selfish. And that he would be taking in my daughter too. I said no way I'm not selfish. I'm honest with you. He'd be adding one. I'd be adding four. No way! I make a modest $40,000 a year. We have our routine and our little two bedroom condo, our little car, no wild expenses. I'm not looking for a whole instant family.