With ONE Sore nipple, can you feed on one breast only?

I’m a FTM, my baby is 3 days old, and my right nipple is slightly inverted so baby has not been getting a good latch. Its already starting to get really sore and cracked, and is becoming unbearable to feed with, but my left nipple is fine. I’m still only expressing colostrum, and my right breast (the cracked one) is starting to swell because I skipped 2 feedings on the right. Now my left breast seems like it’s not expressing colostrum anymore, but my right one has damaged skin vessels and I don’t want to aggravate it any further. What do I do? Do I keep feeding on the left breast, or do I need to just suffer through on the right?

My husband helped me self express colostrum on the right nipple and dap drops of colostrum using his finger and putting it to his mouth...but I don’t know how sufficient that