I need advice!

My boyfriend and I have been together for over a year. I live with him and his family, so things have been a bit hectic and we argue at times, but other than that our relationship is okay. Recently, I started going back to church in my hometown. My SO doesn’t go to church and has made some comments about me going. Since going back to church, I’ve become friends with a guy who goes to my church and who I used to have a “high school” crush on. This guy has asked me to hangout a couple times and flirts heavily. I’m not sure, but I think I’m starting to have feelings for him even though I have a boyfriend. I’m not sure what to do. Do I tell my boyfriend or let the feelings go away and move on?