when your SO is not a believer

So I recently accepted Christ into my heart. But I've been engaged to a wonderful man and we are very much in love but he is not a believer. We have talked about it and he has said that he  accepts me and he will even go to church with me, but he just doesn't believe in any higher power. We have already made commitments and such to each other and while it's not right to be equally yoked it also doesn't feel right to leave someone when you ha e made promises to them. 
Can anyone give me advice on how to be a loyal Christian even though he is not ?