doesn't want to mess around every time

my boyfriend had a talk with me today and told me he didn't feel comfortable messing around or fingering me every time we hang out. i was kind of upset just because i felt like i had made him do too much when he didn't want to. i asked him how i would know if he were in the mood and he said he would tell me but idk how i should feel abt his situation. what's your opinion?