Trying really hard

So I've been trying to build my credit because eventually (and hopefully soon) I would like to have a house of my own! At the beginning of the month I received a letter about a debt I owe for breaking a lease two years ago. I was doing so well, opening credit cards, using them, paying them off, making early payments, paying my car loan on time, all of that! Now that I started payments toward this debt, my score is going down! I know it's gonna go down but once it's paid off till go back up, but I just feel so defeated. I'm putting all this effort and money into bettering myself financially and I just don't seem to be getting anywhere 🤔