Baby after miscarriage

My significant other and I were pregnant last year and miscarried at 11 weeks. He's been begging me to try again, but I don't know if I can go through the heart ache again. I have 3 children already. All very difficult pregnancies and previous miscarriages in between. He has 1 daughter that lives out of state who he is only given 2 hours a year to see. I understand why he wants to try again, I just don't know if I'm up for it. Am I being selfish and inconsiderate?