What's wrong with me???

Hello, this is my first post on here and I would like some advice. Recently, 6 months ago, I moved to a new state and about 3 months ago I started dating this guy. It's been going pretty well I think he's kind to me, listens to me, we hangout a lot, and we have a quite a bit in common. I've been all googly eyed over him until about a week ago. I don't know what it was suddenly I felt trapped and scared. My best friend (who's like a sister to me) doesn't really approve of him, but she's never really like anyone I've dated so I don't take it to heart. But I don't know what it is anymore, I feel so detached. I do suffer from depression and chronic anxiety. Works been pretty harsh on me lately with the long hours I don't get me time anymore.

Some red flags to me though is his past. He went to prison during high school for grand theft auto and sometimes his past with drugs bug me. Like dont get me wrong I smoke pot and drink now and then but he's an ex cocaine addict (more or less still is), takes lots of Molly, drinks a lot more than I do, and has recently revealed how he used to smoke meth. The problem is he's still open to trying more. My friend isn't pleased by the way he teases me which I never get too angry about he's just teasing with me I rub it off but I don't think she's sees the side of him that's caring and sensitive like he always is when I'm alone with him. Another thing is how he'll talk about our future moving back to my home state, teases marriage, kids, I know he isn't always serious but that shit scares me I'm only 20. Yes I wanted a more serious relationship but not this quickly! I like taking things slow at first so we can grow together.

I guess my question is am I crazy for feeling some sort of doubt??? He is great my red flags may sound worse than what it's like in person. He is so far the nicest person I've dated he tries really hard to impress me and care for me.