past relationships/intimate moments

My husband won't talk about his past relationships with me which I think is a bit odd. He was married once before and engaged once. He has told me some very general sexual relations stories but hasn't told me the number of women he has slept with. Most recently we were talking about masterbation, he does not masterbate often and does not watch porn. He told me after the divorce and before me he masterbated but I asked him what the fantasy topic was and he told me he would relive past experiences, not fantasize. When I asked him to tell me he said no he would not tell me what he would picture as it would defile our relationship and if I asked 100 men they would all say the same and I am crazy to ask him. I want to know because if he thinks/thought about it is something he enjoys and I want to help in that department. He said, now I have you, I don't need that. But it still makes me wonder and feel curious that he refuses to share. I already have some trust issues and this doesn't help... so... thoughts? Is it wrong to ask about past intimate moments and want to know what he is/was thinking? Would your men tell you?