HELP! Should I be worried?

My little boy is just shy of 6 weeks old and exclusively breastfed! A week after he was born he lost 6oz, at two weeks old he had lost a further 6oz, at three weeks old he gained 9oz, at four weeks old he didn’t gain or lose any weight and at 5 weeks old he gained 6oz, now today at almost 6 weeks old he has gained only 2oz! The health visitor wants me to stop breastfeeding and refer him to a GP, however he has passed his birth weight, he is settled after every feed, has periods where he is alert and happy, he’s recently started smiling and has plenty of wet and dirty nappies!

Should i be worried and take her advice or should I continue trying to stick with breastfeeding?