To be or not to be

Lady
Im having an affair with a man who has 2 children. he's not a married man, but they still have an affair with the mother of the children though they are not together. We are the ones together. Sex life is very active. We only use withdrawal. But every after my period we dont use any contraceptive because for us its the only time we have "mindblowing" sex.. We satisfy each other well. We started out as just having fun because of his situation. And now its hard for us to stop. To separate. To forget each other.