Seeing an older guy

I'm seeing an a guy that is older than me and there is a massive age difference. All my friends think it's wrong and keeps saying he could rape me, I know how it feels as I've been raped in the past. I know he wouldn't do this to me and we're just having sex there's no feelings with him and he has a finance and 2 kids. I'm just asking it is alright to be doing this with him??