Boyfriend said it might not stick

I'll be 18 weeks pregnant tomorrow. My boyfriend doesn't do emotions, doesn't understand and doesn't want to understand emotions. 
Tonight I brought up labor scaring me because I don't have family here to help this time and with the midwives I'm seeing they don't stay in the too. the whole time. I wanted to know he could take me being in labor and maybe not being nice. His response was "why worry about that? You know you could lose the baby tomorrow." 
Am I wrong that those words hurt so bad? I've lost a baby at 12 weeks before... I am always scared...