Should I have sex with him?

I have a 6 year old daughter, and me and my daughters father were never actually "together"  and we've always had a bad relationship with one another. For some years how his younger brother has been trying to get with me. Out of respect, I havent done anything with him. But lately I have been wanting to... He's very sweet, and caring etc. but I just don't know if I should cross that line...what do you guys honestly think?