paranoid now

When me and my boyfriend first got together he said he had a threesome fantasy but could never do it because he's too self conscious but it was just the idea, ok cool then during sex he would say about how he'd love to watch me with another girl (never bothered me because it was just dirty talk while horny and drunk) then he admits he had one with 2 girls when he apparently walked in on them getting off? To me this seems he's done something and trying to imagine it with me!! Anyhow during a drunken night during sex we were talking dirty, threesome mentioned and he says to me "your best friends in there why don't we invite her in if you're serious" now thinking back this isn't the first time he's mentioned something like this! When he was over mine my friends bed had broken (me and my friend lived together while at uni) and he said why don't we let her sleep in beside us? Then when we moved to a new house and waiting for the unit to be delivered for the tv he suggested once again "why don't we all go up into the bed and watch it in there? You can be in the middle" then another night he suggested she slept in beside us or at the bottom of the bed so one night his friend was around and was going to sleep on the sofa I said to my boyfriend "if you're so keen on people sharing a bed why doesn't he sleep at the bottom?" And he flipped his shit at me saying how he doesn't want to because he knows his mate would try something with me etc. Double standards!! Now I'm paranoid that anytime I'm out of the house or out of the room him and my friend could be doing things behind my back etc. I'm so confused on what to do. I know my friend wouldn't do anything but then I think in the back of my head she gets around men like there's no tomorrow (been with 15 in a year)