bleeding in pregnancy

I'm 21 weeks pregnant with my second baby and from 17 weeks I've been having a little bleeding every now and again. It starts off red then goes pink then brown then stops for a few days and starts again. Its happened 4 times now and I don't know if I should be worried about it or not. I've rang the midwives at the hospital and they never seem too worried about it, they just say to keep an eye on it. I had my 20 weeks scan and everything is fine. Any advice? Has anyone had the same happen to them?