Married and lonely

My husband and I have been married almost 3 years, I have been off the pill over 3 years as we spoke and agreed to start ttc before we got married. Ever since we got married I feel like our sex life has just completely disappeared and I think it's down to the fact my husband doesn't really want to start a family with me. Everytime I try bringing up kids and babies he becomes distant and changes the subject, we've had sex 3 times in 18 months and I really don't know what to do or where to turn, it's not just that we don't have sex but we don't hold hands, kiss or cuddle or have any intimacy. The last 3 months have been really difficult and I've found myself thinking if I want this relationship to work, kids have always been a deal breaker for me and being almost 28 I feel that my time is running out and don't know what to do. I know anyone reading this will say it's obvious and I need to sit down and speak about our problems together but surely if I'm feeling this lonely and unloved he knows and should want to at least talk it out and try to fix it but he just avoids it and me and acts normal. It's driving me crazy I have 2 weeks off work and tried to plan a mini break away for us both to try and get closer and work through some problems but he turned it into an argument about money and refused to do anything. Now I'm sat alone in the house thinking everything over and starting to resent him and my choices, I feel so lonely and really have no one to turn to, I know this isn't the right relationship to start a family in but I can't see staying for another year or two that things will change and keep thinking I will get to the point where it's too late to have my own family.