He told me to lose weight or else...

I've been up and down weight wise for YEARS. My SO met me at a time where I was and am UP. We been together for 5 months now, I've explained to him throughout the course of our relationship my struggle with my weight as well as my family history of weight related medical conditions as obesity, diabetes, hypertension etc.(all to which l do not have) I've shown him recent pictures of myself when I was "down" (15-20lbs lighter 8-6months ago) during the spring/summer. Since seeing the pictures he KEEPS referring to them and making comments saying how good I look then and how I "need" to get back to looking like that. Tonight he told me I need to lose weight or else... I didn't ask "Or else what!?" (Which was my immediate reaction) Honestly I was just too hurt and shocked and didn't want to have a conversation through those feelings 
( I know all too well how conversations can turn ugly when speaking through emotions) I plan on discussing it with him further tomorrow with a cooler head and a lighter heart. Right now I don't know how to take it: is he concerned about me and using an 'ultimatum' as motivation or just being vein and shallow jerk!? 
He's been in my life for 5 months not 5 years and haven't seen the transformations I've made and make (both physically and mentally) I ALWAYS gain weight during fall/winter months and lose A LOT of weight in the spring I lose what I've gain and then some only to put it back on come fall (that's the cycle that is my struggle) I've explained this to him numerous of times I thought the pictures will help him understand Now I'm feeling that it made matters worse... He wants me to snap my fingers and lose 20lbs which isn't happening
Is he being a jerk or just being a man?! What say you and what would you do!? Any and all opinions are appreciated