Nipple irritation

Is nipple irritation common? I'm not breastfeeding, never had a kid, no where near that time in my life. Just happens sometimes when I don't have a bra on, I'm assuming its just irritation from the fabric or something. I've talked to a few friends who say it's happened, but it's so annoying. I'm very hesitant to put anything on them....don't even know what I would put on them but mine are very sensitive. Any suggestions/comments/ideas?