Falling for someone else?

As of late my SO and I have been TTC, and have not yet been successful. I've had a  miscarriage in our last attempt and it really took a lot out of me. As of late, we've been arguing more than usual and I just find myself more so depressed with life. However the one time I don't find myself feeling this way is with this guy I work with. I cannot deny that I find him attractive, but he's also incredibly charming and funny. But I feel horrible that the fact that I get little butterflies from another guy than my bf of 5 years. Any advice?