Itchy Outer Vagina

For the past couple weeks I've had this itch on just one part of my labia, a very small itchy spot. I went to the doctor over a week ago and he gave me an antifungal medication to take saying that the inside looked fine, just the outside looked a bit swollen and irritated. After taking the meds (it was just one dose) it felt better after about 4 days but I had sex with my husband Wednesday night and now it's feeling itchy again. 
The doctor said it could be because I went off BC last month and the change in hormones can cause problems in the vagina. 
Anyone ever experience anything like this or have advice?