Cant come without vibrator?

So i masturbate with a vibrator and can cum fine, if i or my boyfriend finger me however nothing happens. I can do it for 20mins and not even be close to orgasming. I tried to masturbate with my vibrator and stop just before i orgasmed and then use my fingers but as soon as i stopped the vibrator my orgasm went completely and i couldnt again, is this something i cant change or is there something i can do about it??