Would you be suspicious?

Husband went down to out hometown two days ago and didn't come home until 2am. This morning he woke up and told me he was really sick. He came home from work at lunch and had me make him lunch and tea and was complaining that he was going to bed as soon as he got home because he was so sick so wouldn't be able to help me with bedtime for our two kids under two tonight. Fine whatever. An hour after he goes back to work he calls me and tells me he's miraculously feeling better and he's going back down to our hometown (2 and a half hours away) to see his friend and "work on his truck" and wouldn't be home til really late. Am I wrong to be annoyed? 3rd time in the last week and half he's driven down there for random reasons and no real explanation. When I told him that he better be good (implying he better not be cheating) he was like "I'm always good" and made it clear he was pissed I would even say that. I'm so lonely because he's never home, always out with friends and I feel like I'm always home alone with our babies. Am I right to feel weird about this? He says I'm being dramatic.