Tired of Gold digger statements!!!!

I've been with my Fiancé for a little while now and EVERYTIME I start talking to friends or other people about him they assume that I am with him for his money. 😒 I am so tired of feeling like I have to prove myself to damn people about whether or not I really love him. He pays all the bills expect I pay for my car and insurance. I work too! No I don't make as much as him but I bust my ass with 10 hour days 5 days a week. His family comes from money as his dad has a high end job with a major company. But that doesn't mean that's what I care about. I'm tired of having to list the reasons I'm with him so people don't think I'm using him for money. I need advice! What can I do to stop this!?