Am i wrong?

So I've been with my bf for almost a year now he has these married couple friends. He's been friends with them for like 4 years. Well I never cared for his friends but ive always tolerated them never rude always welcomed them. Long story short they experimented with other ppl and they keep trying/hinting my bf try with the girl now also. Me nd my bf can't seem to see the same thing. He's upset that I'm not comfortable with him going over to their house. I rather when they wanna visit they come over. Yes he agreed not to be alone with her ever but its just I'm still uncomfortable with him being alone with even tho the husband will be there it doesn't mean anything I mean I u let your wife be with your brother what's gonna stop from letting her try with his friend. So am I wrong for feeling the way I feel?