Small penis...

I was with a guy with an above average penis for 2 years, and now I'm with someone with a small one. He doesn't give me any pleasure and it keeps popping out. It really drives me crazy but I don't want to tell him because I might hurt his feelings. What to do?