How much do you know about your husband's past?

My husband and I have been together for 6 1/2 years (5 married). He's never really opened up about past relationships except for one in high school. I knew about that one because we are now all friends. 
But this weekend, while cleaning out old files and photos, I discovered letters from an old girlfriend of his from about a year before we even met. He wasn't keeping them for any reason; I don't even think he knew they were in there. 
When I was asking him about the relationship (because there were some questionable things), he clammed up. All he said was this woman was a fraud so I know she must have hurt him bad. 
I started asking about other relationships and he completely shut me down.  Should I be worried or just forget it because it's the past?
By the way, I didn't date much before we met so I don't have much of a past and he knows that.