Help I’m in a weird state

I am in a weird state and don’t know what to do. Yes I want to lose my virginity to my boyfriend but I also don’t want to I’m allergic to latex (yes I know they make latex free condoms) I had some not so good things happen to me in a pervious relationship but I am a survivor I’m not on birth control at all and I keep getting made fun of by people because of my age and that I still have my virginity (I’m 19 almost 20). Before anything gets too crazy yes my boyfriend is willing to wait on me and is very respectful but every time something happens to where we get a bit heated I freak out and literally have a panic attack and I feel like it’s my fault that I can’t do anything I know sex doesn’t make a relationship better I just feel awful about leaving him like that and freaking out when we get to a certain point we have talked about a couple things I trust him completely but I just get really nervous to try anything yes he calms me down and tells me it’s ok and everything’s alright he understands but I just feel awful about it I don’t know what to do.