Anxiety advice

So my boyfriend and I dated in 7th grade. We broke up with a big fight and he told his mother some things about me and she used to hate me. Earlier this year in 8th grade we dated again but he was forced to break up with me because his mom hated me. During the end of the year me and him got close yet again and hung out at his house once school ended, I at the time was in a relationship but he was single. We both sat in this one chair kind of cuddling. He ended up telling me he liked me and I knew I still loved him from the old relationships we had so I told him I liked him. We continued to cuddle but some things lead to another and we were making out at one point. So I broke up with my boyfriend and me and him decided to date. But recently every time I go to his house I get anxiety and I don't know why, I get scared to go over to his so I always don't want to. And then we get the fact that the only way we can hang out is if I go to his house because my dad is crazy about me dating at my age. He wants to hang out with me a lot but I'm still scared to hang out because of my anxiety and I don't know how to get out of it. Any advice?