Anal sex?

So me and my boyfriend have thought about having anal sex recently and decided to give it a try. I've always wanted to try because I would sometimes finger myself back there and it felt good. So a week ago we decided to try it out and got the lube. It HURT. It was like a burning sensation and he barely got the tip in so we stopped, and the pain went away really quickly. Fast forward a bit and we decided to try again. I made sure to relax and we got all the way in. But it felt weird the whole time. I kept feeling like I had to go to the bathroom during even though I didn't need to. I held on but after a while we stopped because he wasn't feeling it either. I was wondering if there is anything I can do to help with the weird feeling? Do we just need to keep trying? I want to try again but the feeling just kills the mood for me.