Bf gone for the night

My boyfriend is off for a construction job and he won't be coming home until the following night (if I'm lucky he could be staying until Sunday). I shouldn't be worried since he's working but when I said are you going to miss me he said no not really kind of glad I get out with a jokingly attitude so I jokingly responded what are you gonna be doing the night? Being with other women and he flat out answered maybe then smiled. After that conversation I couldn't sleep right last night and woke up feeling sick to my stomach. I'm unsure of if I should be worried or not. We have no way of communication for the night and I'm stressing out about this. Should I be stressed? What should I do he already left about 45 mins ago…