I dont I know about this.....

Ive been with my boyfriend for 4 years now we lost our virginity to eachother. We havent been having sex as we normally did 5-6 days a week now we only have sex 2-3 days a week...mainly because he works alot. But a subject has came up that im really not sure about.... he wants to have a threesome. Yeah we have talked about it while having sex just to spice it up... but i didnt actually think he was serious... i want to have the experience in my life but then again im afraid it will ruin our relationship.. what if its not just a one time thing? What if he goes behind my back :/... this is my worst fear.