Scared to sleep over my boyfriends house

So this may sound dumb but I have really bad anxiety/panic disorder and I generally hate sleepovers so much as I normally have panic attacks at them and it's embarrassing because I don't want to leave in the middle of the night. My boyfriend, who lives about 2 and 1/2 hours away, wants me to stay the weekend at his house (we're 18 and both of our parents are okay with it). I really want to go because he's stayed at my house before and I'm going to have to stay at his sooner or later but I'm really scared I'll get too nervous and have a panic attack. As dumb as it is anxiety is about getting sick at other people's houses if that helps. Any advice?