Really need advice! Anyone please

I've been with my boyfriend for 3.5 years- we are both each other's first for everything and now we are completely different. He doesn't have any ambition for his future, he makes plans with his friends before me- if we do have plans he'll either be late or he will easily forget about me. When we talk we are very open and talk about everything- I feel like I'm one of his friends sometimes as he will always swear when he talks to me- and I can't stand that. Tomorrow I'm leaving to go overseas for 6 weeks, he didn't want to travel with me (we both have different holiday styles) which doesn't bother me at all- but this afternoon/tonight he was suppose to hang out with me and sleep over- but he had his friend come over and they both started drinking and obviously can't drive now. I drove to see him before hoping he would come home with me, but no- too busy watching the cricket. All these things happen and nothing will ever change, I hate these things and I don't know why I've stuck around for so long and put up with it. I just can't break up with him and I don't know why!! Plus we've just started building a house which is making me feel even more guilty about my feelings, but I don't know what to do.