Is it ever too late to go back to college?

I never went to college because every single thing I wanted to do my family shot me down and said I wasn’t clever enough and I’d just be wasting my time and not to go. I ended up not going and getting a job in a pharmacy but I got severely bullied by the older women there so I quit after that I was unemployed for a year the worst year of my life i was so depressed and now I’m just in dead end jobs going no where. I feel like my life is pointless and I’m by doing anything worth while. I really want to be a religious education teacher but it will cost me 5000 to go back to college but I also need a job to pay rent and bills. Is it to late and I should just stick where I am or is it worth the risk? Had anyone else gone back to college