I messed up.. and now I'm freaking out..

My fiance and I were trying to get pregnant for a few months when some things happened between his family and I and we split up for about 3 months. Right before we started discussing getting back together because all this was dumb, I had gone on a few dates w someone and slept with him once. We used a condom and I was sooo not worried. About a week later I slept w my ex/ now my once again future husband and then a few weeks later I'm pregnant. I'm extremely excited, but a small part of me is so worried. I was closer to my predicted ovulation when I made my big boo boo but didnt use any protection with my man who has healthy swimmers.. am I freaking out for no reason? P.s. my hubby knows about what went down w the other guy.. I'm not hiding my fears about this from him.