Poly-Relationships (Updated)

Guys, I desperately need some advice. My boyfriend just told me that he's always wanted a poly-relationship, meaning me, him, and another woman. The thing is, that's the last thing on Earth I want. Should I be worried to tell him how against it I am? I just don't know how to feel right now. My heart is racing a mild a minute.

Update: So I told him how poly isn't my thing and how it's never been. I told him how I don't mind him having friends, but relationship-wise he's mine. And he told me how he loves me more than anything, and couldn't live without me. He also said he's not completely sure if us not having a poly-relationship would bother him, because he's never done it before. Personally, I think it's just a fantasy of his. But thanks for the advice, ladies. I desperately needed it. I love him more than anything and he's my first boyfriend, so it wouldn't exactly be easy to just leave him.