Very suspicious of my husband

We moved to a new town two years ago for my husbands job, hours away from our family and friends. Since we've been here I've had our second baby and with two under two my life is pretty hectic, I'm a SAHM. My husband made a group of friends at work. He's always been a social butterfly but this has seems to have gone to the extreme. He goes out with these friends at least 5 nights a week straight after work and doesn't come home until 1:30 or 2 in the morning even if he has work at 7am the next day. He goes in camping weekends twice a month for three days and because I have two very small kids I can't go and I get left at home to do everything for them both. Don't get me wrong, my children are my life and I'm so grateful to have the opportunity to stay at home with them but it's been very lonely. My family and friends live so far away and my husband never invites us to go with him. Tonight he came home for 20 minutes, ate the dinner I made, left and it's almost 2am and he's still not home. We've argued about it at length many times and he's told me if I bring it up again he's going to divorce me and take the kids. I'm suspicious he may be having an affair because I have no idea what goes on at this "friends" house or who is over there when he goes. He's mentioned there was girls casually and that he's smoked cigarettes with his friends fiancé and her female friend but then changed the subject instantly when he realized I was upset about it. Am I right to be concerned? I feel so lost. I hate being this person who has to worry whether or not I'm being cheated on and he calls me crazy if I ever ask him if he's cheating. What would you do?