Does not having sex turn off guys?

I have a problem with guys whenever I tell them "sex is always going to be off the table". I know it sounds silly to wait till marriage and all but that's just my decision. Whenever I tell them that, they seem to dift away and somehow I feel like that's all guys ever want these days, is sex. Is that a bad thing to not want to have sex simply because I'm not ready and I want to wait till marriage?