Stress and exhaustion effects on ttc?

Audrey
I just started a new job. I'm in training for now and I guess I could describe it as being quite physical to me (always standing up, also crouching and lifting things) and mostly I think it's stressing me out because things have to go really fast and it's very noisy. 
I'm okay doing the job (I don't take an immense pleasure doing it but I can manage) but as soon as I'm out of there I just feel a huge crash like all my energy is gone, I feel dizzy, just want to lie down. I feel like all my happiness gone, don't feel like talking or joking. Basically I'm not myself at all and I feel so bad because the thing I love the most is taking care of my SO, taking care of our home, and be "his sunshine when he comes home" (his words not mine)
And I hate how tired I feel and I hate how I can see in his eyes that he clearly sees that I'm not myself since I started working there and I end up crying in his arms because Im exhausted and my body hurts.
And I keep thinking this job would never work if I was pregnant so i'd stop when I'd get a positive. But then idk, would this stress and exhaustion also affect my body enough that it'd be harder ttc? I've been feeling so low that we haven't had sex and I mean that's okay for now -still sad but okay- because I'm not in my fertile window... But what would be the effects in the long run?
My SO just says he wants me to be happy and fulfilled and that this job is clearly not doing it and I could find something else or just stay at home and we'd manage. The job was never really a thing I thought would last because having a baby is the number one priority and I want it to stay that way. The job was just something to earn a bit of extra money while ttc because we can never know how long it will take to get a positive. 
But I don't know, I mean there's this big part of me that's telling me I don't want to go back. But there's the other that tries to rationalize and say that it's because I'm in training that it feels this hard and I could just bear it and stick to the plan. 
I feel so lost and tired