he's 20 years older. please help

Cee
I'm 31 with no kids. My boyfriend of 2 years is 50 with 2 kids. I am madly in love with him to the point that I feel like God is punishing me by giving me someone I can't have. I have always had these huge plans to move to California and marry an amazing guy and have 3 kids and an amazing career. But he is putting one kid through college and the other one will be in college in three years. He says he wants kids with me but I know he won't be able to afford it. He should be retiring in 15 years, not taking care of more kids. How do I find the strength to leave the perfect man? What if no one is ever good enough for me ever again? Time is flying by and I am not ready to suffer the large hole my heart is going to have if I leave him.