Need advice

I feel pathetic going to social media to figure out this problem but it is what it's come down to.
So I don't speak to my "father" anymore. I found out last year he's not my real dad. Skip ahead - My parents split after he was caught cheating on my mom. 
 Haven't talked to him in 2 weeks, and I saw him today and all he said was "damn you are gaining some weight"
Let me remind you, I'm 5'6, just hit 130 pounds.  By no means am I large like he makes me sound. Anyway,
I tell myself I'm not getting big but why do I let it get to me? My family's really athletic and due to me and my mom moving I left my old school and didn't do any sports.
Please someone reach out and help me a little bit