I just had a conversation with my husband I never really wanted to have

I told him I'm ready to give up on trying for a baby we been close to having children but I keep losing them and the thought sometimes wake me up out of my sleep and say not everyone will become a parent... I been feeling this way for sometime now he thinks we should keep trying but I'm breaking down and it's not good for me mentally I hate this feeling but I need to let it go and move on and just enjoy us...