Being too thin affects fertility?

I read online that women who have a BMI of 19 or bellow (considered underweight) might take 4 times longer to get pregnant than 'normal' women. I am 164cm and I weigh between 50-51 kg. I eat well, I'm not starving myself at all... I just don't gain weight! This gives me a BMI of between 18.6 and 19. Should I be worried? Is there anything I can do? Is this even true?