Irregular heartbeat during sex

Sometimes while having sex, my heart starts to beat irregularly and it often beats WAY too hard. It doesn't happen every time, probably 1 in 5 times and I have not noticed a pattern as to what sets it off.
It scares me, I'm often too afraid to finish when it happens because I'm afraid my heart will explode!
Any advice? Has this happened to anyone else? Should I see a doctor?