Feeling insecure...please help

*sorry this is long*

So....when my hubby and I met, I knew that he had recently broken up with his girlfriend/Baby mama that he had been with for 6 years...in fact, I was the one that helped him get through that way before we started dating. that had been a seriously toxic relationship, with a lot of lying and violence and cheating on both ends, and it took him a while to get over it. Once we did start dating, 6 months later he broke it off to fix things with her for his daughter's sake. Within a month, he came back to me after she burned him again. We mended things, got back together, and have been together ever since, with NONE of the drama that I've heard from him and others went on when they were together. Well, tonight at dinner, his mom and I started talking about hair. And she mentioned how one time my hubby had dyed his hair a certain color and his ex had thrown a fit about it Because everyone was complimenting him on it and she didn't want him getting attention like that, and she thought only gay guys dyed their hair...

Here's the dilemma...he and I had been dating for 4 months when he had dyed his hair. He had asked me what I thought, and I had told him I loved it, because it looked really good on him.

Now I'm beginning to wonder if he and her had ever actually broken up, or if they were together the whole time we were dating that first 6 months...we had lived 2 hours apart, so it would have been easy for him to hide it from me...

I guess it doesn't really matter, he's not with her anymore, I know that for a fact, but it just really has been playing on my mind ever since...

He says I'm the first person he's ever truly loved...and yet I feel like at first, I was just someone he used...and I don't like that feeling at all.

How do I shake this feeling off? When we got back together we agreed the past doesn't matter; and I don't want to fight over something that happened over a year ago. I just don't know what to do with this...