Am I an asshole for this?

I (21) have a childhood friend (21), our parents are best friends, and she also has a friend who is older - like 45 maybe, and is a total weirdo. He’s a real life family guy Quagmire and is just weird to me. They drink a lot and, a few times, unexpectedly bought coke and did it in front of me. I am currently a single mom with little help to a 10 month old and 2 year old, work full time on a night time job, and need to buy a car, so I’m busy lately. My days are usually stuck balancing trying to stay up all day to spend time with the littles and sleep before also being up all night. Neither of them have kids, work full time, etc. Lately, they want to hangout more than we did before. When I was hanging out with them before, I was immature and was partying a lot to cope with abuse and things that I was going through. I also stopped drinking because my family are alcoholics and I realized I was headed that way too. But I got my head on straight and realized my priorities and I feel like that’s fair. But I feel like they don’t get it. They say they do, but they don’t seem to. They always try to blow my phone up and if I try to say no, they try to keep pushing. Tonight was the first night I’ve had off in 6 days and they even came out to my house anyways and said I “had no choice” so I let them sit there because they weren’t getting anything I said. Then said “sucks you wouldn’t come out here” “you made us do this and then wouldn’t even see us?”. I liked hanging with them but I just feel like it’s getting too much. I know it’s probably not the greatest idea to get involved with drug users either.