I feel like I disappoint him during sex.

Alright so every time I've had sex I can't orgasm without playing with my clit and I feel like every guy I've had sex with gets disappointed when they notice I'm playing with it. Like they're disappointed they can't please me without the extra touch. Does anyone else feel like they disappoint their man or can any men say whether they feel disappointed or don't care?