"Casual Friendship" - my guy won't commit

I have a relationship with my flat mate, but he doesn't want to date. He says he has feelings for me, and we hang out a lot, watch tv, eat together, go shopping, gym together etc. He also hates the word relationship, and has labelled it as a "casual friendship". The rules are that we can both sleep with other people, but we have to tell each other when it happens. He's a great guy, and I really like him. I'm absolutely terrified of him sleeping with someone else, and he knows. He also says that even if he does be with someone else, its a one time thing and he'll come back to me anyway. Has anyone else had something like this? How did you deal with it? Was your time together still relationshipy even though you weren't together? (when we're alone it is.) 
Thanks!