Skipping antibiotics for an asymptomatic UTI?

Two weeks ago, I noticed mild burning while urinating. This was my only symptom, went to the doc and she said it was probably just irritation from sex. It went away, and came back briefly for a day or two last week. I have no urgency, pain, fever, etc. The burning hasn't occurred for 4-5 days. I had a urine culture done on Friday and my doctor said that at first it didn't seem to be anything but eventually something grew (e.coli). He prescribed me Bactrim but I know it's a heavy duty antibiotic and since I feel fine and have a kick ass immune system (not to say I'm invincible or anything, I don't play around with serious illnesses or infections) so I'm kind of thinking about not taking it, the side effects sound awful. Am I playing a dangerous game by doing this? I do know of the risk of kidney infections but how do I know it didn't just clear out on it's own. 

I've been drinking so much water and testing myself daily with an Azo at home UTI test (leukocytes seem to be decreasing, nitrites are not present- they haven't been present in any test in the past two weeks)