Scared have sex again

So my boyfriend and I frequently have sex. The last time we did it we both agreed that the sex felt weird and just wasn’t the same. We kept it opened minded about everything heard each other thoughts about it all. Then we hanged out again. We were both got scared to have sex again. We both even said that. He knows how I feel about all this. We made sure tell each other everything about this. I feel like something wrong, or something wrong with me. I know I shouldn’t think like that. But for some reasons I am. Just question why I’m I so scared to have sex with him again? Like I shouldn’t be, but I am. He wants hang out again, but I refuse go his house and hang because I don’t want him try do it and feel like something wrong with him. Which, there is nothing wrong with him. I just idk why I am scared now. Should I be scared? I guess I feel like I’m scared I don’t want the connection we have now ruined or anything.