Judgmental boyfriend

I've been on a really strict diet. I have lost a total of 110 pounds so far. I've been off my diet for about 3 weeks..my boyfriend makes comments sometimes saying "you don't need that" when I want to get Starbucks...and just stuff like that. Today he noticed my jeans were loose. He said "are you loosing weight?" I said yes and he said "how? You've been drinking and not on your diet.." I thought it was a little rude...sometimes he makes jokes about my weight...he says he's kidding but it's not funny. It's a sensitive subject for me. He says he's just being supportive..but I think he's being rude a lot of the times...is it me? Do I need to lighten up? Any advice? I don't like getting upset over his comments..I've told him this and it doesn't seem to help...  :( not sure what to do anymore..