Is it ok to reject a job offer?

My boss approached me two months ago and said that there a position on a different team that his boss thinks me and another girl on the team would be perfect for, it’s a sure a thing. I applied for the job, interviewed with the hiring manager everything went well till the both of us was not considered for the job. My boss was put off that they considered another person since they weren’t suppose to interview anyone else (we are all on the same team). Then another person on the team quit and the hiring manager did end up taking one of us, which wasn’t me. Flash forward 3 weeks later another person quit and now my bosses boss offered me that job. I was told the hiring managers are offering me the position . I’ll would start as soon as I accept. I told him since this was so messy last time, that I wanted to think on it till tomorrow. Would you consider being on a team where you are not their first choice or even second choice and the processors on the team keep are quitting left and right? I really don’t want to accept the offer, but I’m afraid that it will hurt my career. Does it hurt your career to reject a job offer on a job that they ultimately rejected you for lol.

I didn’t want it originally I was doing them a favor for applying it. More money, but a lot more stress and I have a vac in a month— that I’m afraid they’ll be stubborn about.