He's annoyed when I cry?

I'm overly sensitive sometimes when it comes to the things my boyfriend says (I'm 20 he's 21), and I cry more than I should. And lately when I cry he just looks annoyed and sighs before asking me what's wrong... I know I should toughen up. But should I be concerned too?