Help: first time sex worries

I feel so conflicted. I've been dating my boyfriend for a while now and the conversation has come up about when we are going to have sex. We both really love each other and it's something I want to do. The only thing is, he's a virgin and I'm not. When I had sex before in my previous relationship it really changed the aspect of the relationship and I know that I shouldn't blame my boyfriend for past problems. But I can't help myself but to worry that things will change. I want to have sex with him but my mind is giving me mixed signals. Advice?