Is this wrong?

So lately I've been thinking a lot about having sex with women. I'm really happy in my relationship and I've been with him for 4 years so I'm not sure why I've even been thinking of it so much. He's only the only person I dated since leaving high school and we got together when I was 20 now we have a 1 year old. I guess the point of the whole ramble is that I'm feeling really guilty about thinking of this so much and I feel like it's wrong of me