Dating him because he lost weight

my best friend & I are juniors now. We've known each other since 7th grade he used to like me but I didn't & he was chubby back then. I was dating someone halfway to the end of 8th grade year up to 9th. I dated a guy in the beginning of 9th grade year & then dated my third boyfriend from the end of 9th up to the beginning of my junior year. I had gone w my best friend to his military ball in 10th but as friends only since both of us were pursuing different people (he still had feelings tho). Throughout those 2 years he lost weight & is very slim/tall now. At the beginning of this school year, he kept asking me out & I kept saying no because I didn't want to ruin our friendship & he asked me for a chance. just one. so I did. but now everyone, my mom included keeps accusing me of dating him only because he lost weight. I've grown to like this boy a lot. However, I feel guilty because they keep telling him that & they're telling me too. He asked me if I'd prefer him fat or skinny & I said I'd prefer him funny because his sense of humor was why we became best friends in the first place. I feel like breaking up with him because it hurts me that they're telling him these things & he'll probably believe that's why I put him off for so long. Idk what to do..