Venting, replies with advice would be great

I need help :/ my SO and I have not been together for very long (4 months but we've known each other longer than a year). I fell in love very quickly and so did he. He lives away at school and he comes home as often as he can and I visit him as often as I can. All of that is great and we have a great connection and feel comfortable around each other. 
When we started dating he was a virgin. About 3 weeks after he asked me to be his girlfriend and try a long distance relationship with him we ended up meeting halfway to hangout and see each other and stayed in a hotel. Previous to this we had not done anything more than kind of feel each other up and just fall asleep together. This sounds so silly but he's religious and had never really been with a girl in any intimate way before. So that night  we were at the hotel we had a lot of fun. We went on a date night in town and met up with some old friends and then when we were on our way back to the hotel he asked me if we could stop at the pharmacy and get condoms. I said yes and we did. He told me I'm not sure if we will but I just want to be ready incase we decide to and I told him that was completely fine and then at that moment told him I had been taking birth control for my endometriosis. Anyway he lost his virginity with me that night but it was all entirely up to him. I asked him if he was positive he was ready and he said he was. 
So fast forward to last weekend when I went to stay with him we were having sex at night when we went to bed. Afterwards I wanted to go again and he told me he was tired and wanted to sleep. I said ok that's fine so we laid down and talked for a little and I was having a hard time falling asleep. I look over and he's on his phone playing games and it kind of just annoyed me and I was like I thought you were tired? And he said I just didn't want to anymore and that kind of upset me more because it felt like he was telling me I'm not attracted to you anymore. So I kind of just went to my side and closed My eyes but he could tell I was upset and I told him when you say that it makes me feel like you're not attracted to me. And he told me then "no I'm totally attracted to you. You're beautiful. It's just that we don't HAVE to have sex everytime we see each other" and so I said ok that's fine but if you didn't want to why didn't you just suggest we watch a movie and cuddle instead of making out with me and instigating it. So whatever we agreed we didn't need to have sex every night we are together. 
So tonight, he came home and we were at his place and no one was home so he was getting kind of horny and what not but I'm on my period so I wasn't feeling it. Later on in the evening to kind of shut him up we ended up just playing around which I didn't mind. And afterwards he sighed really deeply. I asked him what was wrong and he kind of mumbled and said "I didn't want to do this tonight" and I just got so confused because he was the one with the boner when all we were doing was watching a movie. I asked him and told him he needed to be more upfront with me and he said he would but then he was the one that said he thinks we maybe moved too quickly doing that since we started only 3 weeks into us officially being together. Personally to me it didn't seem so quick because I fell for him and I love sex. 
I just don't know what to do because we obviously can't take back having sex