Hair damage- what am I doing wrong?

I've got naturally very fine and thin curly hair. I don't over wash (maybe twice a week because I gym), I condition more often than I wash, I wash with a sulphate free shampoo.

I don't heat style my hair, I don't dye my hair and I don't wear hats or caps or have my hair up (I clip it loosely when I go to gym).

I do an oil mask once a week and I take iron (I'm anemic) as well as a hair vitamin.

Yet my hair is so damaged. I've got those white little spots at the ends of my hair and tons of split ends. My hair is also very brittle.

What am I doing wrong that my hair is like this? I'm trying to grow it out after cutting it dramatically short last year so I'm only going for trims once every couple of months.

Update: I chopped half my hair off yesterday. Screw it, it will grow back.

I'll get a silk pillow case as suggested, my hair is too short now to clip or tie up.

My vitamin levels are fine- I go for blood tests yearly because of a genetic thing in my family and they do full bloods every year. Everything is fine other than my iron which is always on the low side.