Gym problems

My bf && I constantly fight about me not going to the gym. He goes to the gym everyday! && as for me I really don't go I probably go like twice a week && that's if I feel like it or if I have time. He is very fit and me on the other hand I'm not fit at all. I sometimes believe that he doesn't like the way I look because of the remarks he says sometimes. I know I've gained weight within the time we've been together and I've tried so many different diets but I can't seem to lose weight . Well the question here is, should we be getting into arguments over the gym? I know it sounds silly but I guess now he has given me a last chance because if I don't lose weight or try to go to the gym often then that might be the end of our relationship .