Sexually frustrated

Me and my boyfriend moved in together a little over a month ago. The last week we have had no sex, I tried a few times to initiate but as soon as I tried to actually have sex he gets soft. I have been extremely horny lately I've always had a higher sex drive than him but this time it feels so much worse. I feel awful because I've gotten to the point to where I'm actually fantasizing about another man (a specific one) it was someone I had a fling with years ago before me and my boyfriend started dating but I was far more compatible with him when it came to sex so now I'm going crazy! I love my boyfriend more than anything and he says it has nothing to do with me he's just still adjusting to us living together but I need sex! I need great sex! I would never cheat on him but I feel like tearing my own skin off! What should I do? Taking care of myself isn't enough anymore