What should I do

So me and my boyfriend have been together for almost 4 years and his is my first and sometimes I feel like that is the only reason I'm still with him because he is my first real love and I don't want to see him with anyone else... and I'm afraid if I let him go I won't find out until it's to late that I didn't want to lose him.. we fight a lot sometimes.. I honestly cant remember the last time we had sex.. and he gets mad because all I do is work and sleep and watch TV he says but i work a lot because he doesn't work at all.. I just dont know what to do I love him but I dont want to fin out later that I wasted both our lives